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Foreword 


The  Agency  for  Health  Care  Policy  and  Research  (AHCPR)  is  the  Federal  Govern- 
ment's focal  point  for  health  services  research,  including  medical  effectiveness  research. 
AHCPR  s  mission  is  to  enhance  the  quality,  appropriateness,  and  effectiveness  of  health 
care  services  and  improve  access  to  such  services,  through  a  broad  base  of  scientific 
research  and  the  promotion  of  improvements  in  clinical  practice  and  in  the  organization, 
financing,  and  delivery  of  health  care  services. 

AHCPR  has  a  strong  commitment  to  research  on  quality  of  care  measurement  and 
development  of  quality  assurance  methods.  AHCPR  will  support  research  on  these  and 
related  issues  as  part  of  its  continuing  effort  to  integrate  knowledge  on  the  effectiveness 
of  health  care  into  clinical  practice. 

To  help  refine  a  research  agenda  responsive  to  current  needs,  AHCPR  convened  a 
conference  in  March  1992  to  examine  research  priorities  in  quality  improvement  and 
quality  assurance.  The  conference  provided  a  forum  for  discussion  among  health  care 
providers,  researchers,  policymakers,  and  quality  improvement  and  quality  assurance 
professionals  working  in  both  public  and  private  delivery  settings.  This  volume  contains 
commissioned  background  papers,  presentations,  summaries,  and  recommendations 
from  the  conference. 

We  hope  this  report  will  stimulate  research  and  further  discussions  that  will  improve 
the  science  base  for  developing  quality  improvement  and  quality  assurance  initiatives. 

J.  Jarrett  Clinton,  M.D.,  M.P.H. 
Administrator 

Agency  for  Health  Care  Policy  and  Research 
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Preface 


The  publication  of  this  report  marks  what  we  hope  is  the  beginning  of  a  broad- 
based  and  innovative  quality  assurance  and  quality  improvement  research  program  that 
will  build  on  the  work  already  underway  at  the  Agency  for  Health  Care  Policy  and 
Research  (AHCPR).  As  efforts  to  further  health  care  reform  take  shape,  it  is  clear  that 
measuring,  assuring,  and  improving  quality  of  care  will  be  critically  important  and 
require  a  concerted  research  effort. 

This  report  includes  background  papers  and  commentary  summarizing  the  issues 
raised  at  the  1992  AHCPR  working  conference  designed  to  formulate  a  research  agenda 
on  quality  assurance  and  quality  improvement.  The  commissioned  papers  provide  a  com- 
prehensive overview  of  the  conceptual  and  practical  issues  involved  in  building  a  science 
base  to  support  quality  improvement.  They  include  extensive  references  to  current  litera- 
ture and  descriptions  of  innovative  programs.  The  papers  and  commentary  also  pinpoint 
some  concerns  about  definitions  and  information  gaps  and  highlight  issues  regarding  the 
roles  of  medical  professionals,  payers,  and  health  care  consumers  in  quality  assurance  and 
quality  improvement. 

We  believe  that  the  wide  range  of  experiences  and  interests  articulated  during  the 
conference  are  vital  to  the  creation  of  a  meaningful  research  agenda.  Developing  mea- 
sures and  data  to  generate  better  information  on  quality  of  care  was  stressed  throughout 
the  conference.  There  was  also  general  agreement  among  conference  participants  on  the 
importance  of  investigating  the  effects  of  quality  improvement  and  identifying  which 
mechanisms,  processes,  or  models  "work"  in  different  circumstances. 

There  was  less  consensus  on  a  taxonomy  for  quality-related  activities  and  the  optimal 
relationships  and  contributions  of  quality  improvement  and  quality  assurance.  Similarly, 
opinions  regarding  the  use  of  information  by  health  professionals,  peer  reviewers,  regu- 
lators, payers,  purchasers,  and  consumers  often  reflect  the  distinct  perspectives  of  the 
authors  and  conference  participants.  Nevertheless,  important  research  issues  were  iden- 
tified in  all  areas. 

AHCPR  is  actively  pursuing  a  research  agenda  designed  to  identify  the  most  effective 
ways  to  prevent,  diagnose,  and  manage  health  care  problems  and  to  communicate  that 
knowledge  to  practitioners  and  patients.  The  development  of  clinical  practice  guidelines, 
medical  review  criteria,  performance  measures,  and  standards  of  quality  mandated  by 
AHCPR 's  authorizing  legislation  are  important  tools  for  quality  assurance  and  quality 
improvement.  Related  research  on  methods  and  measures  and  the  effects  of  quality 
improvement  is  important. 

The  conference  also  brought  to  light  the  need  to  expand  the  horizons  of  health  ser- 
vices research  beyond  the  areas  noted  above.  Participants  emphasized  the  need  to  under- 
stand more  about  how  health  care  organizations,  coalitions  of  providers  and  purchasers, 
and  delivery  systems  work,  so  that  the  new  technologies  and  methods  of  quality  improve- 
ment are  introduced  successfully.  Priority  issues  generated  in  the  workshops  include  the 
further  development  and  evaluation  of  information  technologies  and  the  need  to  describe 
and  evaluate  what  is  happening  within  organizations  with  respect  to  health  care  quality 
improvement,  peer  review,  utilization  management,  and  other  mechanisms  to  enhance 
quality  of  care.  Organizational  structure,  climate,  and  leadership  are  potentially  impor- 
tant variables,  as  is  the  external  regulatory  and  reimbursement  environment.  Basic  data 
collection,  descriptive  research,  and  experimental  studies  are  needed  to  strengthen  the 
science  base  for  quality  assurance  and  quality  improvement.  We  believe  this  summary 
report  will  contribute  towards  that  end. 

Linda  K.  Demlo,  Ph.D. 

Director,  Office  of  Program  Development 

Agency  for  Health  Care  Policy  and  Research 
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Executive  Summary 


Introduction 

This  volume  summarizes  research  issues  in  qual- 
ity improvement  and  quality  assurance,  which  were 
examined  in  detail  during  an  agenda-setting  confer- 
ence on  "Putting  Research  to  Work  in  Quality 
Improvement  and  Quality  Assurance,"  sponsored 
by  the  Agency  for  Health  Care  Policy  and  Research 
(AHCPR)  in  March  1992.  The  conference  provided 
a  forum  for  interaction  among  health  care 
providers,  researchers,  policymakers,  and  quality 
improvement  and  quality  assurance  professionals 
working  in  both  public  and  private  delivery  settings. 
Participants  examined  the  state  of  the  art  and  dis- 
cussed how  better  to  structure  research  and  devel- 
opment to  strengthen  the  effectiveness  of  quality 
improvement  (QI)  and  quality  assurance  (QA) 
activities  and  their  responsiveness  to  underlying 
needs  of  health  care  delivery  and  public  policy. 

While  there  are  a  variety  of  definitions  of  quality 
review,  a  working  definition  emerged  over  the 
course  of  the  conference.  The  concept  of  QI  was 
used  to  refer  to  a  set  of  specific  approaches  to 
improving  performance,  as  indicated  by  objective 
measures,  using  techniques  such  as  organizational 
redesign,  cross-functional  management,  processes  of 
continuous  improvement  such  as  QI  teams,  and  a 
focus  on  defining  customer-oriented  system  objec- 
tives. QA  was  used  to  refer  to  internal  and  external 
methods  for  assessing  the  incidence  or  levels  of 
quality  problems  and  assure  that  quality  is  achieved. 

The  conference  focused  on  four  major  themes: 

•  Methods  and  measures,  including  quality  review 
criteria,  performance  standards,  and  ways  to 
improve  the  science  base  in  these  areas. 

•  Information  technologies,  including  automated 
data  resources  and  medical  informatics. 

•  Organizational  issues,  including  relationships 
with  health  care  delivery  systems,  control  and 
responsibility  for  quality  review,  and  resources 
for  QI  and  QA. 

•  Using  quality-related  information,  including  the 
roles  of  patients  and  purchasers  as  their  agents 
(for  example,  consortia  of  employers  that  nego- 
tiate with  providers)  in  acquiring  and  using  qual- 
ity-related information  and  the  influence  of  the 


information  on  both  clinical  decisionmaking  and 
the  behavior  of  health  care  institutions. 

Conference  speakers,  panelists,  and  workshop 
participants  identified  common  themes: 

•  Information  technologies  are  both  a  driving 
force  behind  and  a  major  source  of  concern  in 
evolving  QI/QA  systems. 

•  The  field  of  quality  improvement — total  quality 
management  and  continuous  quality  improve- 
ment— holds  promise  for  major  changes  in  orga- 
nization and  practice,  but  there  is  currently  little 
sound  evidence  of  its  effects  on  either  health 
care  outcomes  or  delivery  system  efficiency. 

•  There  is  a  strong  interest  across  the  practice, 
research,  and  payer  communities  in  retrospec- 
tive and  prospective  evaluations  of  models  for 
improving  quality  of  care.  Evaluations  are 
needed  to  identify  what  aspects  of  QI/QA  meth- 
ods and  models  work  or  do  not  work  in  different 
organizational  environments  and  the  factors  that 
facilitate  or  hinder  their  effectiveness. 

The  priorities  for  research  developed  in  the 
workshops  and  discussed  in  this  volume  generally 
reflect  these  basic  themes. 

Methods  and  Measures 

Improved  methods  and  measures  are  essential  if 
expanding  knowledge  about  medical  effectiveness  is 
to  be  translated  into  better  medical  management. 
Conference  participants  addressed  research  issues 
in  the  following  areas:  quality  review  criteria, 
performance  standards,  and  the  science  of  quality 
assessment  and  improvement.  Key  questions 
requiring  attention  are: 

•  Medical  review  criteria.  What  are  the  best  meth- 
ods for  developing,  implementing,  and  evaluat- 
ing medical  review  criteria?  How  do  national 
and  local  criteria  compare?  How  do  criteria 
developed  from  the  generalist  perspective  com- 
pare with  those  developed  from  the  specialist 
perspective  in  terms  of  acceptability  and  useful- 
ness? What  are  the  best  ways  to  design  and 
apply  clinical  guidelines  and  review  criteria 
across  sites  of  care,  taking  into  account  the  locus 
of  responsibility  for  decisions  and  differences  in 
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the  configurations  of  practitioner  settings?  What 
sampling  strategies  are  most  effective  for  specific 
applications? 

•  Changes  in  practice  behavior.  What  are  optimal 
methods  of  information  feedback  for  both  prac- 
titioners and  systems  to  change  practice?  Which 
elements  of  practice  are  most  amenable  to 
change? 

•  Clinical  guidelines.  How  does  implementation  of 
clinical  guidelines  affect  quality  of  care?  What 
difference  do  guidelines  make  in  terms  of  cost 
and  patient  outcomes,  including  functional  status? 

•  Measuring  quality  of  care.  In  addition  to  data 
from  the  medical  record,  what  other  data  on  the 
process  and  outcomes  of  care  should  be  cap- 
tured, compared,  and  contrasted  to  improve  the 
quality  of  care?  What  longitudinal  measures  of 
quality  of  care  best  characterize  patient  care 
episodes  across  the  complete  range  of  practice 
settings?  What  are  the  strengths  and  weaknesses 
of  profiling  methods  as  QA  tools?  What  is 
known  about  the  validity  of  commonly  used 
measures  of  quality  of  care?  Are  there  methods 
to  improve  the  reliability  and  validity  of  meas- 
ures of  quality? 

Information  Technologies 

Research  priorities  for  information  technologies 
focus  on  availability,  quality  and  specificity  of  data 
for  quality  assessment  and  on  specific  ways  to 
address  the  social,  political,  and  other  barriers  to 
using  information  technologies.  Research  questions 
include: 

•  Automated  information  systems.  How  effective 
are  automated  information  systems  as  tools  for 
improving  the  quality  of  patient  care?  How  are 
the  data  in  information  systems  used,  and  what 
other  data  elements  should  be  included  to 
improve  the  systems,  quality  measurement,  and 
quality  of  care?  How  may  patient-specific  proba- 
bilities and  utilities  be  incorporated  into  decision 
support  systems? 

•  Use  of  information  system  technology.  What  are 
the  factors  that  influence  the  adoption  of  infor- 
mation systems  technology  in  QI  and  QA  pro- 
grams? What  is  known  about  the  roles  of  tech- 
nology, policy,  legislation,  and  public  opinion  in 
solving  the  problems  of  privacy  and  confidential- 
ity of  electronic  records? 


•  Diffusion  and  dissemination.  What  are  the  most 
effective  ways  of  diffusing  QI/QA  technologies 
that  improve  quality  and  dissemination  of  the 
information  derived  from  these  systems? 

Organization  of  QI/QA  Systems 

Research  on  the  organization  of  QI  and  QA 
systems  is  critical  to  incorporate  methods  and  meas- 
ures effectively  into  ongoing  programs.  Relevant 
issues  include  organizational  readiness  and  capacity 
to  accept  and  use  new  methods  and  technologies, 
institutional  and  professional  culture,  and  public 
demands  and  expectations.  Research  also  is  needed 
on  how  different  approaches  to  designing  and 
implementing  QI  can  be  structured  to  work  in 
different  health  care  delivery  settings,  how  responsi- 
bility for  monitoring  quality  is  allocated  among 
internal  and  external  review  entities,  and  what 
resources  are  needed  to  assure  and  improve  health 
care  quality.  There  is  a  particular  need  for  descrip- 
tive and  comparative  information  on  what  is 
happening  in  quality  review  systems  throughout 
the  country,  as  well  as  a  need  for  more  information 
on  State  and  Federal  regulation,  accreditation  pro- 
grams, and  certification  systems.  A  variety  of  ques- 
tions require  research  and  evaluation,  including: 

•  Organizational  structure.  Are  certain  organiza- 
tional structures  able  to  more  readily  adopt  QI 
methods  and  programs?  What  models  (or  deliv- 
ery systems)  employ  the  tenets  of  QI  most  easily 
and  why? 

•  Effects  of  QI.  What  have  been  the  measurable 
effects  of  QI  on  the  cost,  quality,  and  effective- 
ness of  the  health  care  provided  by  organizations 
using  QI?  How  have  improvements  been 
achieved? 

•  Oversight  and  review  of  QI  programs.  What  are 
the  effects  of  oversight?  How  may  an  optimal 
relationship  between  an  external  and  internal 
review  of  quality  of  care  in  a  delivery  setting  be 
identified?  How  does  external  review  stimulate, 
support,  or  subvert  the  QI  process? 

•  Management  variables.  Which  management 
variables  (including  both  professional  and 
administrative  services)  that  affect  how  the 
process  of  care  is  organized  and  how  decision- 
making occurs  also  affect  the  outcomes  of  care? 
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Using  QI/QA  Information 

QI/QA  information  has  been  suggested  to  be  a 
key  to  managed  competition  approaches  to  health 
care  and  innovations  such  as  health  insurance  pur- 
chasing cooperatives.  Particularly  important 
research  questions  are: 

•  Communication.  What  information  about 
patient  experiences  can  clinicians  and  provider 
organizations  use  to  improve  care?  What  are  the 
characteristics  of  patient-provider  communica- 
tion that  lead  to  improved  outcomes?  How  can 
the  medical  encounter  be  structured  to  enhance 
effective  patient-provider  communication? 

•  Use  of  data.  How  do  patients  and  purchasers  of 
health  care  use  data  on  the  quality  and  effective- 
ness of  health  care?  What  are  the  effects  of  that 
information  on  patient  health-related  behavior, 
health-care-seeking  behavior,  anxiety  levels,  and 
satisfaction  with  care? 

•  Informed  decisionmaking.  What  information  do 
patients  and  purchasers  need  to  make  informed 
decisions  about  the  quality  of  care  they  can 
expect  when  selecting  health  care  providers? 
How  can  they  be  well-informed  about  the  inter- 
pretations and  uses  of  QI/QA  data? 


•  Purchaser-provider  partnership.  Are  there 
examples  of  purchaser-provider  partnerships  for 
achieving  QI/QA  goals?  If  so,  what  are  their 
unique  characteristics?  For  example,  is  success 
in  achieving  the  goal  of  demonstrated  savings  for 
the  purchaser  associated  with  such  characteris- 
tics as  providing  information  about  quality  to 
consumers? 

•  QI/QA  and  the  community.  Can  QI  make  a  con- 
tribution to  fostering  integrated  models  of  com- 
munity health  service  delivery?  Does  QI  provide 
common  measures  and  tools  for  linking  all  par- 
ticipants? Can  QI  make  a  contribution  to  the 
community?  How  can  barriers  to  comparison  of 
QI/QA  information  across  organizations  be 
addressed? 

Conclusion 

AHCPR  plans  to  support  research  on  these  and 
related  issues  as  part  of  its  continuing  effort  to  inte- 
grate knowledge  on  the  effectiveness  of  medical 
care  into  medical  practice.  The  recommendations 
found  in  this  volume  will  be  incorporated  into 
AHCPR  programs  and  used  to  establish  priorities 
for  new  research,  demonstrations,  and/or  evaluation 
initiatives. 
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Presentation 


National  Health  Policy,  Quality  Research, 
and  Total  Quality  Improvement 

Helen  L.  Smits,  M.D. 


National  Health  Policy 

I  will  begin  with  my  version  of  the  list  of  prob- 
lems in  our  health  system:  access,  cost,  the  hassle 
factor,  and  manpower.  The  first  two  items,  access 
and  cost,  are  universal.  If  you  ask  knowledgeable 
individuals  what  is  wrong  with  our  health  care  sys- 
tem, the  consensus  seems  to  be  that  cost  and  access 
are  our  two  most  serious  problems. 

Hassle  factors  and  manpower  appear  on  the  list 
as  somewhat  eccentric  additions  of  my  own.  By  has- 
sle factors,  I  mean  all  the  elements  that  can  make  it 
extremely  unpleasant  to  be  either  a  patient  in  the 
U.S.  system  or  to  practice  medicine  in  it,  especially 
in  that  supposedly  valued  area  of  practice,  primary 
care.  If  you  doubt  that  we  are  unusual  in  this  area  I 
would  urge  you  to  spend  time,  as  I  have,  talking 
both  to  practitioners  and  residents  of  other  coun- 
tries who  have  been  patients. 

I  have  an  elderly  father  whose  medical  condi- 
tions are  beginning  to  catch  up  with  him.  I  visit  him 
in  New  York  every  month  or  so.  The  first  event  of 
every  visit  is  the  presentation  to  me  of  a  file  of 
Explanation  of  Benefit  forms,  mostly  with  denials  of 
claims.  My  stepmother  is  British  by  birth,  and  she 
finds  this  aspect  of  American  culture  baffling.  We 
are  doing  pretty  well;  the  folder  is  getting  thinner. 
But  requiring  an  84-year-old  man  to  write  four  let- 
ters to  prove  that  he  has  Medicare  Part  B  when  he 
has  faithfully  paid  those  premiums  for  19  years  is 
nonsense  and  a  terrible  waste  of  money.  I  am 
pleased  to  report  that  I  have  now  given  them  the 
greatest  possible  assistance  they  could  have  asked 
for:  I  have  found  him  a  primary  care  physician  who 
accepts  assignment. 

Hassle  factors  include  the  terrifying  implications 
of  access  for  many  working  Americans.  I  doubt  that 
many  at  this  conference  are  either  self-employed  or 
employed  by  small  companies.  I  have  the  great  priv- 
ilege of  living  in  a  small  town  in  Connecticut  where 
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there  are  many  self-employed  individuals  and  many 
who  work  for  small  businesses.  These  people  share 
a  common  sense  of  gloom  about  the  potential  loss 
of  health  insurance  or  the  potential  escalation  of 
premiums  into  an  unaffordable  range. 

Finally,  manpower.  This  is  not  a  fashionable 
worry  these  days,  given  the  depths  of  our  recession 
and  the  fact  that  many  women  have  returned  to 
nursing  or  have  increased  their  work  time,  thus 
reducing  the  shortage  we  faced  only  a  few  years 
ago.  But  the  reality  is  that  the  out-year  projections 
are  frightening.  This  is  particularly  true  in  areas 
other  than  nursing  such  as  laboratory  technology 
and  x-ray  technology.  We  suffer  as  health  care 
employers  from  the  pervasive  problems  that  plague 
the  American  educational  system.  In  my  own  city  of 
Hartford  it  is  not  clear  who  will  be  available  in  10  or 
15  years  for  the  technologist  jobs,  given  the  weak- 
ness of  our  high  schools. 

Total  Quality  Improvement 

There  are  many  who  know  more  about  total 
quality  improvement  (TQI)  than  I  do.  What  I  have 
to  offer  is  a  personal  perspective  that  comes  of  hav- 
ing wrestled  with  the  issues  as  a  member  of  the 
board  of  the  Joint  Commission  on  the  Accreditation 
of  Healthcare  Organizations  (JCAHO),  as  the 
director  of  a  hospital  that  is  beginning  to  lay  out  a 
plan  for  TQI,  and  as  a  participant  in  a  number  of 
training  sessions  and  seminars. 

To  me,  the  most  exciting  thing  about  TQI  is  not 
its  radical  qualities  but  the  opportunity  it  gives  us  to 
rework  activities  we  are  already  engaged  in  and 
understand.  First,  TQI  requires  many  of  us  to 
acquire  skills  that  we  used  to  leave  to  the  manage- 
ment engineers.  You  must  fully  understand  the 
process,  and  you  are  given  straightforward  tools  to 
facilitate  your  understanding.  TQI  couples  those 
tools  with  the  concept  that  front-line  employees 
should  solve  the  problems  presented  by  their  work. 
If  your  objective  is  to  correct  a  problem  in  same-day 
surgery,  such  as  delays,  you  ask  the  clerks,  doctors, 
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and  nurses  to  work  together  to  figure  out  what  is 
wrong,  and  you  offer  them  the  opportunity  to  fix  it 
themselves. 

Another  attractive  element  of  TQI  is  the  con- 
cept that  hospitals  should  satisfy  their  customers, 
both  internal  and  external,  and  that  satisfaction 
should  be  rigorously  measured.  If  you  work  in  the 
billing  office,  part  of  your  job  is  to  be  sure  that 
people  understand  the  bills  they  receive  and  are 
satisfied,  as  much  as  anyone  can  be,  that  the  billings 
are  fair  and  honest.  In  addition,  you  must  satisfy 
internal  customers  who  need  information  about 
volume  and  revenues.  If  all  the  elements  of  an 
organization  are  one  another's  customers,  and  if  the 
services  that  we  provide  to  one  another  must  be  of 
the  highest  quality,  then  the  way  we  treat  one 
another  will  change  radically. 

Finally,  at  the  heart  of  TQI  is  the  concept  that 
quality  can  be  both  measured  and  reported.  Aggre- 
gated reports,  just  like  financial  reports,  can  be 
provided  to  the  hospital  board,  and  more  detailed 
information  can  be  provided  to  the  chief  executive 
officer  and  individual  department  heads.  These 
reports  on  quality  constantly  remind  us:  Where  is 
it  this  month?  Where  was  it  last  month?  Where  do 
we  want  it  to  go? 

While  this  sounds  ideal,  it  has  not  yet  been 
proven  in  health  care.  There  are  some  hospitals  that 
are  several  years  into  the  process  and  enthusiastic 
about  the  results,  but  I  have  yet  to  hear  of  any 
hospital  that  can  report  results  as  dramatic  as  the 
Plexiglas®  manufacturing  plant  in  Connecticut  that 
has  been  deeply  involved  in  TQI  for  10  years.  They 
have  documented  improvement  in  efficiency  of  sev- 
eral hundred  percent  and  a  documented  drop  in 
employee  turnover  and  employee  dissatisfaction. 
You  have  to  believe  that  TQI  will  continue  with  a 
track  record  like  that.  We  have  not  reached  that 
point  in  health  care,  although  we  are  working  hard 
to  get  there. 

Barriers  to  TQI:  A  Research  Agenda 

I  recommend  that  the  design  of  a  TQI  research 
agenda  begin  with  an  examination  of  the  barriers  to 
TQI  in  health  care.  First,  there  is  the  problem  of 
rigidly  defined  professional  roles.  TQI  will  not  work 
if  the  team  decides  that  the  nurse  could  obtain  some 
insurance  history  while  she  is  collecting  medical 
information  and  the  nursing  office  vetoes  the  idea 
on  the  grounds  that  insurance  is  "not  a  nursing  job." 


Or,  conversely,  it  will  not  work  if  the  team  wants  to 
train  clerks  to  take  blood  pressures,  but  the  idea  is 
vetoed  because  that  is  a  nursing  job.  The  ability  to 
exchange  roles  and  the  ability  of  front-line  workers 
to  define  and  control  what  they  do  is  a  very  impor- 
tant part  of  TQI. 

The  Plexiglas®  plant  requires  that  everyone  on 
the  assembly  line  be  trained  to  do  all  jobs.  The  com- 
pany vice  president  who  serves  as  on-site  manager 
also  can  do  everything.  The  work  groups  have 
decided  to  organize  their  lives  to  suit  themselves. 
The  plant  is  organized  into  two  permanent  12-hour 
shifts,  and  work  teams  are  unique  to  one  shift  or  the 
other.  The  day-shift  group  has  decided  that  it  likes 
to  change  jobs  frequently,  so  every  4  to  6  weeks  day 
workers  move  on  to  a  different  task.  Night-shift 
employees  seem  more  set  in  their  ways;  they  shift 
tasks  only  every  quarter  or  every  6  months.  It  may 
sound  like  a  small  difference,  but  it  provides  an 
enormous  sense  of  control  to  the  workers  on  both 
shifts.  I  am  still  looking  for  the  hospital  where  nurs- 
ing staff  can  handle  task  assignment  so  differently 
from  one  nursing  unit  to  another. 

Formal  external  regulation  is  another  serious 
barrier  to  TQI.  The  more  specific  and  detailed  the 
regulation,  the  more  difficult  it  is  to  enforce  and  the 
larger  the  barrier  it  presents  to  improving  the  insti- 
tution. One  of  my  favorite  examples  is  the  Federal 
requirement  that  nursing  home  medication  orders 
be  rewritten  every  30  days.  I  know  how  to  get 
them  written  approximately  every  30  days:  hold  a 
doctor's  coffee  hour  on  the  second  Tuesday  of 
every  month,  and  ask  that  orders  be  rewritten  at 
that  time.  All  of  us  schedule  by  months  and  think 
in  months;  a  "second  Tuesday"  is  easy  to  block  out 
in  a  schedule  and  is  likely  to  work  well.  The  prob- 
lem is  that  the  regulators  will  find  the  home  in 
violation  because  in  some  months  the  time  between 
orders  will  be  5  weeks  instead  of  4.  Does  it  matter? 
Is  the  time  spent  enforcing  a  30-day  rule  really 
worth  it?  I  do  not  think  so. 

The  problem  of  rigidity  of  regulation  is  far 
worse  in  nursing  homes  than  in  hospitals,  but  we 
certainly  find  it  in  hospitals  as  well.  Hospitals  suffer 
particularly  from  the  difficulty  of  conflicting  rules 
and  requirements  for  different  payers.  The  more 
time  you  have  to  spend  remembering  to  contact 
this  health  maintenance  organization  on  day  3, 
and  that  one  on  day  5,  the  less  time  you  have  to 
work  on  the  internal  problems  that  lengthen 
hospital  stays. 
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Confidentiality  of  the  oversight  data  is  another 
interesting  issue  that  to  date  has  received  too  little 
discussion.  In  a  Plexiglas®  plant  there  are  no  prob- 
lems inherent  in  letting  everyone  know  what  the 
error  rate  is  or  how  many  units  shipped  have  been 
rejected  by  purchasers.  But  telling  the  cleaning 
personnel  that  the  infection  rate  has  risen  in  the 
operating  room — an  outcome  they  have  good 
reason  to  know  about — is  a  very  different  matter. 

I  am  personally  sensitive  on  this  issue  because  I 
run  a  State-owned  hospital  and  no  one  bothered  to 
exclude  us  from  Connecticut's  Freedom  of  Informa- 
tion Act  (FOIA).  This  means  that  a  plaintiff's 
lawyer  with  a  case  related  to  infection  could  use 
FOIA  to  try  to  obtain  internal  information  on  our 
infection  rates.  As  a  result,  we  are  very  cautious  in 
distributing  information  and  very  careful  to  keep  it 
within  conventional  quality  assurance  (QA) 
channels.  I  think  this  issue  is  relevant  for  any  health 
care  institution.  If  you  send  a  note  to  800  employees 
saying  they  improved  in  one  area  last  month  and 
got  worse  in  others,  the  chances  are  that  sooner  or 
later  those  sheets  will  end  up  in  a  lawyer's  hands.  I 
believe  this  problem  deserves  much  more  attention 
and  discussion  than  it  has  yet  received. 

Finally,  every  hospital  interested  in  TQI  has  a 
problem  in  deciding  how  to  work  with  the  medical 
staff.  They  are  part  of  the  system  and,  at  the  same 
time,  not  part  of  it.  You  cannot  require  them  to 
attend  training  courses  or  take  the  time  to  work  in 
groups  to  solve  the  problems  they  complain  about. 
If  TQI  goes  forward  without  them,  there  will  be 
problems,  but  at  the  same  time  it  is  difficult  to  get 
them  to  be  as  involved  as  they  should  be.  I  believe 
that  TQI  offers  an  opportunity  to  rework  the  doc- 
tor-hospital relationship  in  a  way  that  is  beneficial 
to  both  sides,  but  it  is  not  going  to  be  easy.  Working 
towards  that  end  is  going  to  take  time  and  money 
for  all  of  us. 

Research  and  TQI 

How  does  quality  research  support  TQI?  First, 
we  need  measurement  tools.  In  many  instances,  an 
instrument  that  works  well  for  research  purposes — 
such  as  a  survey  form — can,  with  care,  be  adapted  to 
monitor  day-to-day  performance.  But  the  technol- 
ogy transfer  to  ordinary  use  does  not  just  happen; 
questions  about  the  form's  length  and  the  training 
needed  to  administer  it  must  be  considered  first. 
Interestingly,  the  greatest  efforts  to  produce  practical 


measurement  tools  have  been  carried  out  by  for- 
profit  consultants  with  results  that  are  not  made 
public.  I  think  we  need  more  public  effort  to 
develop  workable  measurement  tools  for  common 
use  and  more  reporting  of  results  on  all  of  the 
instruments  now  in  use  so  that  institutions  can  make 
sensible  decisions  about  what  to  use  when. 

We  also  need  help  in  defining  professional  roles 
and  examining  the  rigidities  that  separate  them. 
Whether  or  not  we  use  TQI  tools  for  improvement, 
we  need  help  in  looking  at  which  tasks  within  a  hos- 
pital or  a  home  health  agency  can  be  shared  in  and 
identifying  those  barriers  to  smooth  function  we 
have  created. 

Next  is  the  problem  of  affordable  data  systems.  I 
am  constantly  astonished  by  the  amount  of  money 
hardware  and  software  vendors  expect  to  make  in 
this  country  from  the  movement  toward  TQI.  I 
spoke  last  fall  at  one  of  the  annual  conferences  of 
the  International  Society  of  Quality  Assurance.  The 
attitude  of  those  who  work  in  other  countries,  even 
relatively  wealthy  countries,  is  very  different  from 
ours.  They  are  interested  in  doing  the  most  they  can 
with  the  data  they  have;  they  want  to  maximize  use 
of  their  relatively  simple  collection  systems  before 
they  add  more  data  points. 

I  have  always  been  a  great  admirer  of  the  basic 
research  behind  diagnosis-related  groups  (DRGs) 
because  of  the  conceptual  elegance  involved  in 
designing  a  classification  system  that  uses  only  data 
elements  already  collected  and  of  known  and 
relatively  high  quality.  The  designers  were  well 
aware  that  they  could  have  made  a  "better"  system 
with  more  information,  but  they  chose  the  more 
challenging  route  of  restricting  themselves  to  what 
was  available.  As  some  of  the  far  more  expensive 
competing  systems  have  begun  to  show  their  weak- 
nesses, I  trust  the  underlying  intellectual  elegance  of 
DRGs  will  be  better  understood  and  appreciated. 

We  must  consider  conflicting  reporting  require- 
ments. The  JCAHO  now  has  indicator  data  sets  and 
is  asking  hospitals  to  collect  new  information.  Moni- 
tors are  also  coming  from  other  directions  and  ask 
for  similar  kinds  of  information.  I  think  an  effort  to 
develop  a  small,  simple,  and  high  quality  data  set  for 
clinical  evaluation  purposes  is  essential. 

Finally,  we  do  not  know  very  much  about  using 
aggregate  data.  I  have  worked  with  the  Connecticut 
Hospital  Association  on  a  project  to  employ  State- 
wide data  on  a  voluntary  basis  to  help  all  hospitals 
in  their  QA  activities.  There  are  few  road  maps  the 
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various  committees  can  follow  to  help  make  large 
aggregate  data  bases  useful  for  local  quality  efforts. 
We  are  now  well  into  an  era  when  researchers  are 
doing  remarkable  things  with  data  aggregation  and 
data  use.  The  more  practical  question  is  how  to 
aggregate  and  display  data  so  institutions  can  better 
understand  themselves. 

Research:  Other  Needs 

Practical  tools  for  quality  improvement  are  by 
no  means  the  only  need  we  have  to  help  us  improve 
our  institutions.  Whether  it  is  called  efficacy  or 

outcomes,  we  need  to  know  more  about  what 
works  and  what  does  not  work.  There  has  been 
much  attention  in  the  press  recently  to  the  fact  that 
several  famous  individuals  have  been  treated  for 
prostate  cancer.  Several  of  them  fall  into  age  groups 
that,  in  another  country,  would  have  led  to  a  treat- 
ment protocol  which  called  only  for  watchful  wait- 
ing. We  tend  to  treat  in  this  country;  we  tend  to 
treat  aggressively,  and  we  tend  to  treat  at  high  cost. 
I  can  do  a  great  deal  to  ensure  that  my  health  center 
is  screening  for  prostate  cancer,  entering  patients 
into  protocols,  and  administering  drugs.  But  I 
cannot,  in  an  individual  institution,  answer  the 
question  of  whether  we  are  benefitting  men  over  a 
given  age  when  we  treat  the  disease. 

Having  said  all  this,  let  me  go  back  to  the  begin- 
ning and  say  that  I  do  not  think  quality  research 
should  focus  only  on  what  we  are  doing  and  how 
well  we  are  doing  it.  Like  the  drunk  in  the  old 
story,  that  puts  us  in  the  position  of  looking  for  the 


keys  under  the  light  when  we  ought  to  know  they 
are  not  there. 

We  work  in  a  troubled  health  care  system.  I 
know  some  very  nice  and  very  hard  working  people 
in  my  town  who  have  no  health  insurance  and  live 
in  terror  because  of  it.  We  have  to  fix  this  system 
soon,  and  we  have  to  ask  research  questions  that 
help  the  people  who  want  to  fix  it.  I  would  expect 
that  anyone  here  could,  if  called  upon,  stand  up  and 
tell  us  in  some  detail  about  small  area  variation  in 
medical  practice  and  what  it  proves  about  the  ten- 
dency of  surgeons  to  do  surgery.  I  also  expect  that 
almost  all  of  you  think  of  this  as  a  tool  to  measure 
institutional  or  physician  performance. 

Small  area  variation  is,  interestingly  enough, 
also  a  way  to  look  at  social  need;  it  is  possible  to 
examine  hospital  discharges  in  New  York  City  by 
zip  code.  What  you  see  is  wide  variations  in,  for 
example,  hospitalization  rates  of  children.  These 
variations  come  not  from  physician  practice  or 
hospital  greed  but  from  the  lack  of  medical  care  and 
even  from  the  lack  of  adequate  housing.  Acute  res- 
piratory diseases — including  acute  asthma — exacer- 
bations of  diabetes,  and  dehydration  secondary  to 
diarrhea  are  the  markers  of  an  inadequate  health 
care  system.  Small  area  variation  can  tell  us  in  detail 
just  how  inadequate  it  is  in  parts  of  New  York. 

Aggregate  data  bases  tell  us  an  unhappy  story 
that  we  have  already  heard  but  neither  loudly  nor 
clearly  enough:  problems  with  quality  once  patients 
are  inside  an  institution  are  not  the  most  severe 
problems  we  face.  Institutions  have  to  keep  improv- 
ing, but  our  entire  health  care  system  must  continue 
to  improve  as  well. 
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Presentation 


A  Perspective  on  Quality  Assurance  Research 

Sheldon  Greenfield,  M.D. 


The  purpose  of  this  presentation  is  to  provide  an 
overview  of  quality  assurance/quality  improvement 
research.  At  the  outset,  it  is  important  to  distinguish 
effectiveness  from  quality.  Many  people  assume 
they  are  the  same,  but  there  is  a  critical  difference: 
effectiveness  is  an  essential  part  of  quality  of  care, 
but  it  is  not  synonymous  with  quality  of  care.  Even 
if  we  know  what  is  effective  and  what  should  be 
done,  it  does  not  mean  it  will  be  done  well. 

Quality  of  care  assessments  go  well  beyond 
effectiveness  in  an  attempt  to  explain  how  things 
were  done.  In  the  case  of  a  procedure,  for  example, 
was  it  done  well;  was  postoperative  care  good;  was 
interpersonal  care  good;  was  care  efficient;  and  so 
forth?  Quality  of  care  differs  from  effectiveness, 
although  the  two  overlap.  Most  people  working  in 
this  field  are  doing  effectiveness  research.  Those 
interested  in  quality  get  less  attention.  It  is  much 
easier  to  get  a  paper  published,  for  example,  on  the 
effectiveness  of  a  drug  or  procedure  than  on  a  qual- 
ity of  care  method  or  quality  of  care  findings. 

This  was  a  very  timely  conference  for  multiple 
reasons:  one,  the  resurgent  interest  in  quality  of  care 
research;  two,  better  research  methods;  three, 
recognition  of  the  need  for  a  comprehensive 
approach  to  quality  of  care;  four,  vastly  improved 
information  technologies;  five,  recognition  of  the 
need  to  change  behaviors;  and  six,  awareness  of  the 
need  for  approaches  that  are  sensitive  to  the  context 
of  care — that  is,  the  setting  where  care  is  rendered. 

We  have  developed  increasingly  sophisticated 
methods  and  approaches  in  the  quality  of  care  field. 
There  has  been  an  explosion  of  research  in  the  field 
of  outcomes,  not  only  to  improve  measures  of  out- 
come but,  more  importantly,  to  appreciate  what 
goes  into  a  good  outcomes  study.  There  also  has 
been  an  enormous  amount  of  work  on  guidelines, 
although  not  so  much  on  the  "downstream  effect" 
— that  is,  what  happens  to  them  after  they  are 
released.  The  research  done  at  RAND  and  by 
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David  Eddy  and  many  others  on  guideline  develop- 
ment has  been  a  key  advance. 

Management  Information  Systems 

As  chairman  of  the  Agency  for  Health  Care 
Policy  and  Research's  (AHCPR's)  Health  Care 
Technology  Study  Section,  I  have  been  involved  in 
the  review  of  proposals  for  many  of  the  major  man- 
agement information  systems  in  this  country.  For 
the  most  part  they  were  funded  in  isolation  or 
certainly  without  having  a  quality  of  care  purpose 
specified.  There  have  been  many  advances  in 
national  databases,  but  the  question  remains:  can 
medical  information  systems  be  customized  to  pro- 
vide the  kind  of  information  on  quality  that  we  need 
so  much?  Some  of  the  most  important  advances 
have  come  through  the  work  of  William  Tierney 
and  Clement  McDonalds  group  at  the  University  of 
Indiana.  They  created  a  medical  information  system 
that  tells  you  more  than  whether  something  was 
done  or  not  (such  as  a  claim),  but  actually  reveals 
the  result,  so  you  can  stage  the  patient,  guide  care, 
and  judge  outcomes  at  some  later  time. 

In  our  Patient  Outcomes  Research  Team 
(PORT)  study  on  diabetes — which  is  taking  place  at 
the  University  of  Indiana,  the  Group  Health  Coop- 
erative in  Seattle,  and  the  Tufts  Associated  Health 
Plan  in  Boston — we  are  attempting  to  minimize  our 
dependence  on  the  medical  record.  Instead,  we  are 
relying  on  the  administrative  data  base  and  direct 
information  from  the  patient.  I  believe  the  medical 
information  systems  to  support  these  activities,  with 
their  potential  to  customize  for  quality,  have  been  a 
major  advancement. 

Behavioral  Change 

It  is  now  recognized  that  although  we  can  assess 
something,  that  does  not  necessarily  mean  we  can 
improve  it.  All  the  cost  containment  literature  has 
now  confirmed  what  we  already  knew:  professionals 
do  not  change  easily.  They  lack  motivation  to 
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change,  and  a  new  approach  to  behavior  change 
among  professionals  is  needed. 

Finally,  there  is  the  issue  of  context  sensitivity. 
Those  of  us  who  have  gone  back  and  forth  between 
different  settings  to  do  quality  of  care  research  have 
a  better  sense  of  the  differences  that  exist,  not  only 
in  the  data  that  are  available,  but  also  the  purposes, 
motives,  and  other  factors  that  affect  settings  in 
which  people  work.  These  factors  must  be  taken 
into  account;  if  you  take  measures  off  the  shelf  that 
work  in  one  place,  it  is  not  surprising  that  they  do 
not  work  in  another.  There  is  now  a  finer  apprecia- 
tion for  sensitivity  to  context  and  the  need  to  tailor 
measures  and  other  ideas  to  individual  situations. 
This  may  run  counter  to  what  many  researchers  and 
innovators  would  like  to  see,  because  when  they  put 
forth  a  method,  they  assume  it  can  be  used  univer- 
sally; that  is  not  necessarily  the  case. 

Quality  of  Care  Research 

These  advances  make  possible  some  quality 
improvement  activities  that  many  of  us  never  imag- 
ined at  an  earlier  time.  I  will  cite  several  studies,  a 
few  of  which  are  related  to  our  own  work  and  some 
that  represent  the  work  of  others.  They  reflect  the 
incredible  advances  in  quality  of  care  research  and 
hint  at  some  developments  that  are  on  the  horizon. 

First  is  the  study  by  O'Connor,  Plume,  Olmstead, 
and  colleagues  (1991)  on  the  mortality  outcomes  of 
procedures,  specifically  mortality  following  coronary 
artery  bypass  graft  (CABG)  surgery.  Crude  hospital 
mortality  rates  between  surgeons  were  adjusted 
using  multiple  variables,  and  the  differences 
remained.  This  was  also  done  for  different  medical 
centers,  with  the  same  result.  The  investigators 
came  to  the  conclusion  that  they  had  chosen  those 
variables  which  themselves  could  cause  death, 
independent  of  the  quality  of  care.  They  were  able 
to  select  these  variables  because  there  are  more 
than  20,000  published  studies  of  CABG. 

O'Connor  and  colleagues  took  things  a  step 
further  and  put  all  the  variables  in  multivariate 
equations;  then  they  tested  the  value  of  each  vari- 
able as  it  predicted  the  outcomes  to  decide  whether 
the  variable  could  be  used  as  a  confounder.  After  all 
the  confounding  variables — that  is,  severity,  case 
mix,  type  of  admission,  age,  sociodemographics,  and 
so  forth — had  been  accounted  for,  it  was  possible  to 
conclude  that  quality  of  care  might  be  the  culprit. 
With  respect  to  mortality  following  surgery,  this  is  a 


model  study  for  choosing  the  right  variables,  doing 
the  right  analysis,  putting  it  in  the  right  context,  not 
overinterpreting  the  information,  and  making  it 
plausible  to  readers. 

Another  example  is  the  so  called  "Six  Hospital 
Study"  in  which  I  was  a  participant.  Three  hospitals 
in  Boston  (Brigham  and  Women's  Hospital,  Massa- 
chusetts General  Hospital,  and  the  New  England 
Medical  Center)  and  three  hospitals  in  California 
(University  of  California,  Los  Angeles;  University 
of  California,  San  Francisco;  and  Stanford)  partici- 
pated in  the  study.  An  early  report  of  this  study 
appeared  in  the  Journal  of  the  American  Medical 
Association  (JAMA)  (Cleary,  Greenfield,  Mulley, 
and  others,  1991).  We  selected  six  common  condi- 
tions and  looked  at  patient  outcomes  for  3  months 
to  1  year  following  hospital  admission  using  func- 
tional and  health  status  data  based  on  Paul  Cleary's 
and  John  Ware's  measures.  Then  we  went  back  to 
the  charts  to  look  for  case-mix  and  hospital  process, 
volume,  and  complication  measures. 

A  brief  look  at  the  results  of  this  study — in 
particular,  the  substudy  on  total  hip  replacement — 
indicates  how  this  information  might  be  used  in  a 
quality  of  care  study.  Patients'  physical  functioning 
in  four  of  the  six  hospitals  at  1  year  following  total 
hip  replacement  for  arthritis  was  one  of  the  outcome 
dimensions.  Physical  function  scores  ranged  from  0 
to  100.  After  first  controlling  for  age,  education, 
marital  status,  and  sex,  the  scores  were  75,  78,  74, 
and  68  at  these  hospitals.  Next  we  used  another 
model  in  which  we  added  baseline  functional  status. 
In  the  third  model,  we  added  comorbid  disease — 
that  is,  all  the  remaining  conditions  affecting  the 
patient  were  summarized  into  a  single  number.  The 
differences  appeared  as  we  changed  models.  For 
example,  originally  there  was  a  statistically  signifi- 
cant and  important  clinical  difference  between  two 
of  the  hospitals.  As  we  put  comorbidity  and  baseline 
functional  status  into  the  models,  the  difference  got 
smaller  and  then  disappeared.  This  demonstrated 
that  we  can  build  models  to  show  what  difference 
quality  of  care  might  make  with  respect  to  recovery 
and  functional  status  but  not  to  mortality.  We  must 
take  care  not  to  attribute  some  of  these  differences 
to  quality  when  in  fact  they  result  from  other  factors. 

As  in  the  O'Connor  study  each  of  these  variables 
was  related  separately  to  outcome,  both  bivariately 
and  multivariately.  The  hospitals  differed  widely  on 
these  factors:  in  one  community  hospital,  50  percent 
of  the  patients  had  major  levels  of  comorbid 
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disease.  In  a  West  Coast  referral  hospital,  19  percent 
of  the  patients  had  major  comorbid  disease.  This 
study  showed  that  we  can  use  functional  outcomes 
of  hospitalization  as  one  measure  for  continuous 
quality  improvement  (CQI). 

Gene  Nelson  and  Dave  Buchanan  at  the 
Hospital  Corporation  of  America  are  conducting  an 
innovative  study  on  acute  myocardial  infarction. 
They  are  adding  a  CQI  model  to  the  process  and 
collecting  outcomes  data  at  two  different  times. 
They  are  looking  at  administrative  processes  as  well 
as  physician  decisionmaking,  for  example,  the  time 
it  took  the  patient  to  get  from  the  emergency  room 
to  the  coronary  care  unit. 

In  the  Medical  Outcomes  Study,  we  examined 
four  factors  (system,  specialty,  utilization,  and  inter- 
personal care)  in  relation  to  health  status  and  qual- 
ity of  life.  The  whole  analysis  depended  on  how  well 
the  quality  of  life  measures  reflected  quality  of  care. 
The  study  had  a  cross-sectional  part  in  which  we 
went  to  the  offices  of  physicians  and  collected  infor- 
mation from  over  20,000  patients  and  their  doctors. 
We  followed  about  10  percent  of  these  patients  over 
a  period  of  4  years.  The  study  was  conducted  in 
three  major  cities,  and  we  included  five  systems  of 
care  involving  more  than  500  providers. 

There  was  great  interest  in  whether  we  could 
develop  the  right  measurements  for  use  in  outpa- 
tient settings  to  assess  quality  of  care.  If  we  lacked 
good  measures  of  outcome  and  case  mix,  we  would 
be  totally  dependent  on  the  medical  record — a 
situation  some  of  us  feel  is  less  than  ideal.  There- 
fore, we  developed  simple  clinical  severity  levels. 
We  compared  these  severity  levels  with  functional 
status  to  see  if  they  had  a  relationship  that  would 
ground  the  functional  status  measures  in  reality. 
Such  a  relationship  would  mean  that,  on  average, 
if  a  person  is  sicker  by  our  clinical  standards,  they 
will  report  themselves  as  sicker.  In  addition,  if  there 
is  a  relationship,  it  also  indicates  that  the  clinical 
measures  are  working. 

We  were  not  looking  for  a  perfect  relationship 
because  then  health  status  would  not  add  anything 
to  clinical  severity  or  vice  versa.  This  can  be  seen  in 
our  diabetes  data;  as  clinical  severity  increases,  gen- 
eral health  perception,  physical  function,  and  role 
function  go  down  markedly,  yet  there  is  a  wide 
standard  deviation.  Between  two  people  with  the 
same  disease  level,  one  has  a  very  good  functional 
status  and  the  other  is  down  on  the  scale,  suggesting 
a  major  role  for  medical  care.  We  have  done  this  for 


a  series  of  diseases.  I  want  to  emphasize  that  one 
of  the  validity  issues  for  measurement  of  health 
status  or  quality  of  life  has  to  do  with  grounding  the 
measure  in  clinical  reality  and  measuring  its  ability 
to  predict  clinical  outcomes. 

Armed  with  these  measures,  we  can  do  some 
interesting  things.  As  reported  in  the  March  25  edi- 
tion of  JAMA  (Kravitz,  Greenfield,  Rogers,  and 
others,  1992;  Greenfield,  Nelson,  Manning,  and  oth- 
ers, 1992),  we  examined  utilization  in  these  systems: 
health  maintenance  organizations  (HMOs),  multi- 
specialty  group  prepaid,  solo  or  single  specialty 
group  pre-paid,  multi-specialty  group  fee-for-ser- 
vice,  and  finally,  solo  single  specialty  fee-for-service 
(traditional)  practices.  We  looked  at  multiple  para- 
meters of  utilization  and  found  big  differences  in 
utilization  between  systems.  There  were  significant 
differences  in  case  mix,  and  we  adjusted  for  about 
20  indicators  of  case  mix,  including  health  status  and 
clinical  severity.  Even  after  adjustments,  hospitaliza- 
tions by  the  solo  fee-for-service  practices  were  still 
41  percent  higher  compared  with  HMOs.  If  we  have 
done  this  right,  we  have  developed  utilization  vari- 
ables that  will  allow  us  to  look  downstream  and  say 
that  the  patients  of  these  physicians  are  or  are  not 
getting  more  for  their  money  than  the  patients  of 
other  physicians.  This  method  can  be  used  as  an 
independent  input  variable. 

The  Massachusetts  Peer  Review  Organization 
(PRO),  in  conjunction  with  the  Cleveland  Clinic, 
Lahey  Clinic,  and  Ochsner  Clinic,  is  beginning  a 
similar  study  on  quality  of  care  for  diabetes  and 
hypertension.  They  will  look  at  various  outcome- 
dependent  measures  and  compare  them  with  the 
regular  PRO  external  review  and  with  claims  data. 
This  area  of  practice-based  outcomes  and  quality  of 
care  research  is  very  promising. 

At  the  Harvard  Community  Health  Plan,  we  are 
combining  outcomes  and  guidelines  in  the  same 
study.  The  doctors  involved  in  this  study  went 
through  a  guideline  program;  the  objective  was  to 
examine  the  degree  to  which  they  used  the  guide- 
line, how  they  used  it,  and  whether  it  affected  out- 
comes. For  example,  if  the  care  was  too  parsimo- 
nious, did  that  have  a  negative  effect  on  quality  as 
reflected  by  outcome? 

After  the  doctors  finished  the  course,  we  used 
the  dyspepsia  guideline  in  treating  patients  and  fol- 
lowed the  patients  to  obtain  outcomes  data.  We  col- 
lected information  3  to  4  months  after  the  patients 
appeared  for  treatment,  so  we  had  good  outpatient, 
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disease-specific  data.  The  levels  of  SF-36  outcomes 
matched  other  national  data  for  outcomes,  so  we 
knew  the  parameters  were  good  in  terms  of  our  col- 
lection of  outcomes  data.  We  examined  adherence 
to  guidelines  and  could  see  how  often  things  were 
done  (for  example,  how  often  patients  were 
referred  to  gastroenterology).  This  gave  us  a  sense 
of  relating  process  of  care  to  outcomes  by  following 
guidelines.  This  serves  several  purposes:  one,  it 
shows  the  value  of  the  guidelines,  and  two,  using 
information  on  process  of  care  provides  insight  that 
outcomes  alone  cannot  furnish.  This,  along  with 
CQI,  provides  a  mechanism  for  the  simultaneous 
use  of  outcomes  and  guidelines. 

Conclusion 

There  was  a  time  when  we  despaired  of  using 
outcomes  to  assess  quality  because  outcomes  were 
considered  dangerous,  unpredictable,  and  beyond 
our  control.  Some  of  the  barriers  to  using  outcomes 
are  being  conquered.  If  we  can  deal  with  these  bar- 
riers, ultimately  we  will  be  able  to  develop  outcome 
standards  that  are  not  muddled  by  other  factors  and 
can  give  us  an  idea  of  whether  quality  of  care  was 
good  or  bad. 

In  theory,  for  every  disease  and  for  every  out- 
come, there  are  certain  processes  that  we  carry  out 
and  there  are  certain  confounders.  Some  say  this  is 
too  complicated,  and  we  can  never  measure  it.  But 
our  colleagues  at  the  National  Institutes  of  Health 


and  in  some  other  places  around  the  world  are 
investing  a  few  billion  dollars  into  mapping  the 
genome.  Maybe  in  15  years  we  can  come  up  with 
outcome  models  for  20  major  conditions,  outpatient 
and  inpatient,  and  be  able  to  make  accurate  assess- 
ments of  quality  using  these  outcomes. 
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Quality  Improvement  and 
Health  Care  Reform 

A  report  by  the  National  Academy  of  Science's 
Institute  of  Medicine  (Lohr,  1990)  confirmed  what 
few  policymakers  acknowledge:  poor  quality  of 
health  care  is  a  major  problem  in  the  United  States. 
After  2  years  of  reviewing  scientific  papers,  con- 
ducting nationwide  hearings,  and  visiting  both 
health  care  providers  and  patients,  the  Institute  of 
Medicine  cited  widespread  overuse  of  expensive 
invasive  technology,  underuse  of  inexpensive 
"caring"  services,  and  error-prone  implementation 
of  care  that  harms  patients  and  wastes  money 
(Lohr,  1990;  Palmer,  1991). 

Poor  quality  often  increases  the  cost  of  care  in 
the  long  run.  For  instance,  delays  in  making  diag- 
noses or  in  starting  treatment  permit  a  patient's 
health  to  deteriorate  until  the  severity  of  illness  and 
its  complications  make  treatment  far  more  expen- 
sive and  less  likely  to  succeed.  Outright  errors  in 
diagnosis  and  treatment  waste  resources  and  may 
even  harm  patients  so  that  their  injuries  require 
further  expensive  treatment  (Johnson,  Brennan, 
Newhouse,  and  others,  1992;  Reynolds,  Rizzo,  and 
Gonzales,  1987).  Failure  to  design  and  operate  safe, 
timely,  and  efficient  systems  for  implementing 
patient  care  can  cause  expensive  errors  and  wasted 
resources  (Berwick,  1989).  Therefore,  improving 
quality  of  decisionmaking  and  implementation  for 
health  care  is  an  essential  component  of  any  cost 
control  program.  The  need  to  control  costs  is  the 
force  driving  initiatives  for  health  care  reform.  By 
implication,  then,  quality  improvement  (QI)  is  a  key 
ingredient  of  a  reformed  health  care  system. 


Dr.  Palmer  is  Director,  Center  for  Quality  of  Care  Research  and 
Education,  Harvard  School  of  Public  Health.  Dr.  Adams  is 
Research  Coordinator,  also  with  Harvard's  Center  for  Quality  of 
Care  Research  and  Education. 


This  report  is  being  published  at  a  critical  time: 
health  care  reform  at  last  is  back  on  the  public 
agenda  because  costs  are  accelerating,  and  Ameri- 
cans are  losing  access  to  the  benefits  of  health  care. 
In  reforming  our  health  care  system,  we  should 
build  the  means  for  QI  into  the  design.  This  charge 
presents  planners  with  many  challenging  questions 
that  can  be  answered  only  through  research.  To 
organize  our  thinking  about  research  on  quality  of 
care,  we  need  a  model  for  health  care  delivery. 

In  the  last  half  of  the  20th  century,  not  just  one, 
but  three  conceptual  models  for  the  health  care 
system  have  emerged:  the  professional,  bureaucratic, 
and  industrial  models.  The  professional  model  exists 
primarily  as  a  means  for  physicians  (and  subse- 
quently other  health  care  professionals)  to  deal  with 
professional  control  of  and  responsibility  for  quality 
of  care,  but  it  also  contains  assumptions  about  the 
production  and  financing  of  health  services.  The 
bureaucratic  and  industrial  models  present  rival 
schemes  for  the  production  and  financing  of  health 
services,  but  they  also  contain  assumptions  about 
control  of  and  responsibility  for  quality  of  care.  Each 
model  is  used  by  its  own  adherents  to  advocate  and 
evaluate  policies  for  health  care.  These  protagonists 
seldom  acknowledge  explicitly  the  model  behind 
their  arguments,  a  situation  that  obscures  the  real 
issues  in  many  health  care  policy  debates. 

We  must  understand  these  different  models  in 
order  to  interpret  successes  and  failures  of  past 
efforts  to  assure  or  improve  quality.  This  is  a  key 
step  in  planning  for  quality  in  a  reformed  health  care 
system.  The  shape  of  that  plan  will  set  the  QI 
research  and  development  agenda.  I  will  review  the 
three  models  and  their  implications  for  strategies 
targeted  to  quality,  discuss  the  contributions  made 
by  quality  assurance  (QA),  and  explore  the  promise 
of  continuous  quality  improvement  (CQI)/total 
quality  management  (TQM).  This  review  will  pre- 
pare the  way  for  the  discussion  of  a  research  agenda. 
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Professional  Model 

Freidson  defines  a  profession  as  a  group  of 
workers  who  control  their  own  work  (Freidson, 
1970;  Freidson  and  Rhea,  1963).  Physicians  control 
their  work  in  various  ways.  First,  a  physician  has 
special  expertise  that  the  patient  wants  but  does 
not  have;  therefore,  the  patient  is  in  a  relatively 
dependent  position,  vulnerable  to  exploitation 
(Pellegrino,  1987).  Members  of  the  medical  profes- 
sion control  entrance  to  their  own  ranks.  Admission 
to  the  profession  is  obtained  by  demonstrating 
acquisition  of  special  expertise  (Freidson,  1970). 
This  expertise  is  not  a  commercial  property;  it  is 
obtained  not  by  purchase  but  by  fulfilling  require- 
ments defined  by  leaders  in  the  profession,  namely 
attendance  at  an  approved  medical  school  and  pas- 
sage of  a  qualifying  examination  (Pellegrino,  1987). 
Third,  control  of  medical  practice  is  exercised  by 
professional  colleagues.  On  graduating  from  med- 
ical school,  new  physicians  mark  their  entrance 
into  the  collegium  by  taking  an  oath  to  accept  the 
responsibilities  of  the  profession  (Pellegrino,  1987). 
These  responsibilities  include  a  fiduciary  relation- 
ship with  the  patient  and  a  responsibility  shared 
with  colleagues  for  maintaining  professional  stand- 
ards. Professional  standards  are  evaluated  by 
respected  peers  (Freidson,  1970).  The  profession 
is  "a  company  of  equals,"  but  respect  is  accorded 
to  colleagues  in  proportion  to  their  expertise 
(Freidson  and  Rhea,  1963;  Goss,  Battistella, 
Colombotos,  and  others,  1977;  Hall,  1946).  Despite 
advice  from  peers,  each  physician  retains  individual 
responsibility  for  making  decisions  on  the  care  for 
his  or  her  own  patients. 

The  primary  professional  mechanism  for  quality 
control  is  the  personal  responsibility  and  integrity 
of  each  physician  (Freidson  and  Rhea,  1963; 
Pellegrino,  1987).  This  personal  responsibility  is 
supplemented  by  the  second  mechanism  described 
above,  voluntary  collegial  activities  among  physi- 
cians (Goss,  1963).  Professional  colleagues  help  one 
another  to  provide  good  quality  care  and  certify 
quality  or  award  credentials  to  demonstrate 
accountability  to  the  public  (Roberts,  Coale,  and 
Redman,  1987).  Within  the  medical  staff  of  a  hospi- 
tal or  group  practice,  collegial  means  to  promote 
quality  are  case  conferences,  review  of  work  by 
peers,  and  lectures  on  advances  in  medical  knowl- 
edge (Roberts,  Coale,  and  Redman,  1987).  These 
are  internal  mechanisms. 


There  are  also  external  mechanisms  for  pro- 
moting quality.  Examples  of  these  that  take  a 
collegial  form  include  the  disciplinary  and  educa- 
tional activities  of  State  medical  and  specialty 
societies.  Specialty  boards,  composed  of  physicians 
certified  in  a  given  specialty,  also  accredit  post- 
graduate training  programs  and  certify  competence 
for  specialty  practice  by  examination  of  applicants 
on  completion  of  training.  In  hospitals,  there  are 
requirements  of  the  physician-dominated  Joint 
Commission  on  the  Accreditation  of  Healthcare 
Organizations  (JCAHO)  (Roberts,  Coale,  and 
Redman,  1987) 1  for  medical  staff  review  commit- 
tees. Among  these  are  physician-credentialing 
committees,  which  may  require  specialty  training 
for  a  physician  to  have  privileges  for  a  particular 
type  of  practice  in  the  hospital. 

The  third  mechanism  for  ensuring  quality  is  also 
external.  State  governments,  as  guardians  of  the 
public  interest,  restrict  licensure  to  those  with 
proven  expertise  in  medicine  (Cohen,  1974; 
Relman,  1985).  Since  State  government  officials, 
like  patients,  lack  special  expertise,  they  turn  to 
physicians  to  help  them  distinguish  competence 
from  incompetence.  Medical  schools,  run  by  physi- 
cians and  accredited  by  the  physician-dominated 
Association  of  American  Medical  Colleges,  train 
entrants  to  the  profession.  The  National  Board  of 
Medical  Examiners,  composed  of  physicians, 
provides  examinations  that  State  governments 
accept  as  demonstrating  competence  for  licensure. 

Licensing  also  serves  physicians'  interests.  It 
gives  them  a  monopoly  and  provides  physician- 
dominated  State  licensing  boards  with  the  means 
to  restrict  the  numbers  of  potential  competitors 
permitted  to  enter  practice.  Originally,  a  State 
license  to  practice  medicine  was  valid  for  a  lifetime. 
However,  the  State  licensing  board  can  withdraw 
the  license  to  practice  from  a  physician  found  to 
be  impaired  or  incompetent  or  to  have  violated 
professional  ethics.  In  all  these  ways,  as  Freidson 
(1970)  notes,  members  of  the  profession  control 
access  to  and  continuance  in  the  profession. 


The  Joint  Commission  on  Accreditation  of  Hospitals  (JCAH) 
was  founded  in  1952;  in  1987  it  changed  its  name  to  the  Joint 
Commission  on  Accreditation  of  Healthcare  Organizations 
(JCAHO).  Hereafter  in  this  text,  the  name  of  this  organization 
that  applied  at  the  time  period  being  discussed  will  be  used. 
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Bureaucratic  Model:  Health  Care  as 
Administering  Benefits 

By  the  1960s,  advances  in  medical  science  and 
health  care  were  occurring  so  rapidly  that  changes 
in  professional  work  became  inevitable  (Starr, 
1982).  These  advances  in  health  care  technology 
made  possible  treatments  that  are  effective  but  often 
expensive  and  dangerous.  The  pace  of  change  has 
become  such  that  knowledge  acquired  in  training  is 
no  longer  sufficient  for  a  lifetime.  Physicians  who  fail 
to  learn  about  new  developments  are  a  danger  to  the 
public;  a  physician  can  lose  competence  not  only 
through  impairment,  but  also  through  failure  to  keep 
pace  with  changes.  To  cope  with  the  explosion  of 
knowledge,  physicians  increasingly  specialize  within 
a  narrow  domain  of  expertise;  even  so,  individual 
physicians  are  hard  pressed  to  keep  up  with  medical 
research  even  in  a  relatively  narrow  specialty.  Not 
surprisingly,  it  was  during  the  1960s  that  the  propor- 
tion of  physicians  in  single  or  multispecialty  group 
practices  began  to  grow  rapidly,  while  the  proportion 
of  physicians  in  solo  practice  began  a  rapid  decline. 
Group  practice  was  advocated  during  the  early  1970s 
as  a  way  to  promote  quality  through  collegial  sup- 
port (Graham,  1972). 

The  organization  of  health  services  is  also 
affected  by  the  rapid  progress  in  medical  science.  As 
more  effective  health  care  technologies  emerge,  the 
variety  and  complexity  of  services  expand.  Intensive 
care  is  provided  for  patients  undergoing  major  pro- 
cedures or  surviving  the  initial  stages  of  catastrophic 
illness.  Continuing  ambulatory  care  is  given  for 
chronically  ill  patients  over  long  periods.  Preventive 
care  also  becomes  an  obligation.  Since  the  medical 
profession  is  splintered  among  specialties,  many 
physicians  may  be  needed  to  take  care  of  a  single 
patient.  Furthermore,  many  other  professionals — 
such  as  operating  room  technicians,  medical  social 
workers,  physical  therapists,  nutritionists,  and  phle- 
botomists — now  share  in  the  care  of  patients.  Many 
types  of  equipment  and  facilities,  as  well  as  the  per- 
sonnel to  maintain  them,  are  essential.  With  so  many 
types  of  personnel  working  together,  communication 
and  coordination  of  shared  responsibilities  become 
the  key  to  quality  care.  Hospitals  and  large  group 
practices  expand  their  operations  to  handle  this 
increased  complexity,  developing  internal  adminis- 
trative systems  for  care  delivery. 

Inevitably,  new  technologies  also  increase  the 
cost  of  care.  Even  in  the  1930s,  the  expense  of  med- 
ical technology  strained  the  financial  resources  of 


patients.  Health  insurance  was  introduced  to  permit 
prudent  persons  to  provide  for  the  possibility  of 
needing  health  care.  By  the  1960s,  the  cost  of  health 
care  had  grown  so  much  that  public  opinion  sup- 
ported Federal  and  State  governments  taking  on  a 
role  as  third-party  payers  for  health  care  for  the 
poor  and  elderly.  A  new  type  of  organization  also 
emerged  that  combined  the  financing  and  produc- 
tion of  health  services:  originally  called  prepaid 
group  practices,  these  evolved  into  health  mainte- 
nance organizations  (HMOs)  whose  numbers  grew 
rapidly  in  the  1980s. 

The  model  initially  developed  for  coordination 
and  control  of  health  care  in  this  era  was  bureau- 
cratic. Bureaucracies  are  large  organizations 
employing  full-time  salaried  workers  who  are  hired, 
retained,  and  promoted  on  the  basis  of  how  well 
they  fit  performance  specifications  (Downs,  1967). 
Bureaucracy  is  a  way  to  ensure  an  orderly  flow  of 
complex  activities  in  a  large  organization.  To 
achieve  this,  the  bureaucracy  is  organized  as  a  hier- 
archy; supervisors  control  and  evaluate  the  work  of 
their  subordinates. 

Outputs  of  a  bureaucracy  are  not  evaluated  with 
reference  to  external  markets  (in  terms  of  profit 
over  cost  of  inputs)  (Downs,  1967).  This  means  that 
organizational  strategy  is  not  directed  toward  meet- 
ing customer  demand.  Instead,  and  this  is  a  unique 
characteristic  of  bureaucracy,  work  is  governed  by 
the  use  of  highly  specified,  publicly  disclosed  proce- 
dures that  ensure  impartial  decisionmaking.  These 
impersonal  rules  are  used  to  prevent  political  and 
personal  preferences  from  influencing  the  distribu- 
tion of  services  or  the  career  prospects  of  employ- 
ees. Workers  are  governed  by  these  impersonal 
rules,  which  they  do  not  set.  Bureaucracies  that  pro- 
vide services  apply  impersonal  rules  to  determine 
eligible  beneficiaries.  Bureaucracies  that  contract 
for  delivery  of  services  use  impersonal  rules  to 
select  service  providers. 

How  the  bureaucratic  model  operates.  The  par- 
ticipants in  health  care  are  seen  differently  in  the 
bureaucratic  as  opposed  to  the  professional  model. 
Those  who  are  cared  for  are  not  patients  but  benefi- 
ciaries; individual  physicians  are  seen  as  providers 
or  even  as  "vendors,"  terms  that  also  apply  to  hospi- 
tals and  group  practices.  Hospitals  and  prepaid 
group  practices  are  thought  of  as  institutions  run  by 
administrators.  They  employ  many  types  of  profes- 
sional, technical,  clerical,  and  other  workers.  Physi- 
cians, however,  occupy  a  unique  position.  They  are 
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not  usually  employed  by  the  institution  (Freidson 
and  Rhea,  1963;  Goss  and  others,  1977;  Hall,  1946), 
except  in  staff  model  HMOs,  yet  they  need  the  insti- 
tution and  its  staff  in  order  to  care  for  patients.  The 
institution  also  needs  physicians,  because  only  they 
can  admit  patients  and  order  tests  and  treatments. 

In  this  model,  the  public  interest  is  still  guarded 
by  State  government,  but  the  new  actors  are  third- 
party  payers:  State  and  Federal  governments  and 
private  insurers  (including  HMOs).  The  payers  rely 
on  physicians  to  determine  who  should  get  the  ser- 
vices that  are  the  benefits  of  insurance.  For  govern- 
ment, in  its  role  as  payer,  a  new  relationship  is  set 
up  with  physicians. 

Hospitals  and  HMOs  meet  three  conditions 
inherent  in  the  definition  of  bureaucratic  institu- 
tions: they  are  large,  they  have  salaried  staffs,  and, 
until  the  mid-1980s,  they  were  not  subject  to  market 
forces.  However,  physicians'  work  still  remains 
outside  the  bureaucratic  mold.  Medical  sociologists 
such  as  Freidson  (1970)  and  Mechanic  (1976)  honed 
their  theories  of  professionalism  by  studying  the 
conflict  between  the  professional  and  bureaucratic 
models  as  the  bureaucratization  of  medicine 
proceeded.  The  professional  model  for  physicians 
calls  for  supervision  of  professional  work  by  influ- 
ence and  advice  of  expert  physicians,  with  the  indi- 
vidual physician  retaining  personal  responsibility 
(Goss,  1963;  Goss  and  others,  1977;  Mechanic, 
1976;  Perrow,  1965).  However,  for  administrative 
issues  (which  are  regarded  as  "nonprofessional"), 
physicians  reluctantly  accept  bureaucratic  authority 
(Goss,  1963;  Goss  and  others,  1977;  Perrow,  1965). 

To  minimize  the  clash  between  the  professional 
and  bureaucratic  cultures,  hospitals  and  HMOs 
usually  have  parallel  and  separate  hierarchies  for 
professionals,  such  as  physicians  and  nurses,  and  a 
separate  hierarchy  for  general  administration.  How- 
ever, within  the  physician  hierarchy,  the  professional 
model  still  applies  to  medical  decisionmaking: 
individual  physicians  retain  authority  over  and 
responsibility  for  the  care  of  their  own  patients. 
Each  hierarchy  within  the  institution  administers  its 
own  rules,  but  none  has  the  authority  to  set  or 
change  the  rules  of  the  other  hierarchies.  Changing 
rules  and  procedures  to  improve  quality  is  therefore 
difficult.  Recommendations  for  change  must  be 
passed  up  through  the  ranks  to  the  top,  and  negotia- 
tions for  change  must  be  conducted  among  the 
heads  of  hierarchies.  When  many  suggestions  for 
small  changes  converge  upon  busy  department 


heads  they  lack  time  to  address  them.  The  problems 
that  prompted  suggestions  for  change  are  remote 
from  the  experience  of  the  department  head  and, 
accordingly,  get  low  priority.  The  rigidity  of  the 
parallel  hierarchies  in  a  hospital  thus  creates  inertia. 

Payment  in  the  bureaucratic  model.  When 
health  care  became  too  expensive  for  the  individual 
to  pay  out  of  pocket,  health  insurance  was  intro- 
duced to  take  over  a  function  that  used  to  be  solely 
the  business  of  patient  and  physician.  Hence  the 
term  "third-party  payer."  A  final  feature  of  the 
bureaucratic  model  was  set  in  place  when  third  par- 
ties paid  hospitals  on  a  cost-plus  basis,  and  HMO 
premiums  and  individual  physicians  fees  were  set  in 
nonmarket  conditions. 

With  the  advent  of  Medicare  and  Medicaid  in 
the  1960s,  government  became  the  major  third- 
party  payer.  Federal  health  care  policy  influenced 
events  outside  the  Medicare  program;  because 
Medicare  is  the  single  largest  payer  for  health 
care,  policies  developed  for  this  program  are  often 
adopted  subsequently  by  the  private  sector.  The 
Medicare  and  Medicaid  programs  forced  govern- 
ment to  be  concerned  with  who  gets  services  and 
who  can  provide  them.  As  the  bureaucratic  model 
requires,  the  Federal  Government  sought  imper- 
sonal rules  to  decide  which  providers  could  receive 
public  payments.  Government  used  JCAH  accredi- 
tation standards  to  decide  which  hospitals  were 
"deemed  certified"  as  Medicare  vendors  (Affeldt 
and  Walczak,  1984).  In  effect,  JCAH  requirements 
for  internal  QA  thus  ceased  to  be  a  voluntary  pro- 
fessional activity  and  became  part  of  a  bureaucratic 
regulatory  mechanism  for  excluding  bad  hospitals. 

When  bureaucracies  allocate  human  services, 
they  apply  impersonal  rules  that  determine  eligibil- 
ity for  benefits  rather  than  focusing  on  determining 
and  meeting  the  needs  of  individual  clients 
(Mechanic,  1976).  Inevitably,  the  Medicare  and 
State  Medicaid  programs  applied  this  approach  to 
payment  for  health  care.  Since  government  officers 
were  not  competent  to  determine  eligibility  for 
specific  medical  services  without  physician  exper- 
tise, they  relied  upon  physicians  to  do  it.  For  exam- 
ple, physicians  were  viewed  by  policymakers  as 
applying  impersonal  rules  to  beneficiaries  to  decide 
who  should  receive  surgical  procedures  and  admis- 
sions to  hospitals  for  medical  care.  The  beneficiary 
was  seen  as  a  passive  recipient.  Although  indepen- 
dent office  practices  are  not  themselves  bureau- 
cracies, physicians  in  private  practice  also  were 
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coopted  for  this  traditional  bureaucratic  role — that 
is,  certifying  that  beneficiaries  could  receive  non- 
hospital  care  and  authorizing  payment. 

Government  policymakers  tend  not  to  appreci- 
ate the  fact  that  medical  decisionmaking  does  not 
follow  simple  impersonal  rules.  Rather  it  requires 
much  complex  branching  logic,  with  decisions  often 
made  in  the  face  of  great  uncertainty.  Furthermore, 
medical  decisions  have  to  be  based  on  the  patient's 
preferences,  and  patients  therefore  must  be  active 
participants  in  medical  decisionmaking.  What  is 
"right"  will  vary  from  one  patient  to  another. 

Strategy  for  quality  in  the  bureaucratic  model. 
In  the  1970s,  QA  became  mandatory  for  hospitals 
through  two  mechanisms.  First,  as  mentioned 
above,  JCAH  accreditation  had  become  virtually 
mandatory  as  the  means  for  a  hospital  to  be  certi- 
fied as  a  Medicare  provider  (Affeldt  and  Walczak, 
1984).  In  1974,  the  JCAH  added  to  its  accreditation 
standards  the  requirement  that  hospitals  operate 
internal  QA  (Affeldt  and  Walczak,  1984;  Roberts, 
Coale,  and  Redman,  1987).  This  was  a  radical 
departure  from  the  professional  model  of  physician 
autonomy.  The  rationale  cited  by  the  JCAH  was  a 
malpractice  lawsuit  in  1964  which  led  to  the  judg- 
ment that  hospital  trustees  are  ultimately  responsi- 
ble for  the  quality  of  care  given  by  staff  physicians 
(Jacobs,  Christoffel,  and  Dixon,  1976). 

The  methods  mandated  by  the  JCAH  to  assure 
quality  were  derived  from  physicians'  research  on 
quality  measurement  but  were  melded  with  then- 
current  industrial  strategies  for  QA.  A  major  activ- 
ity was  medical  audit,  later  modified  and  renamed 
"patient  care  evaluation."  This  was  a  cyclic,  quality- 
promoting  activity  focused  not  on  individuals  but 
on  problem-solving  for  the  entire  system  of  delivery 
(Joint  Commission  on  Accreditation  of  Hospitals, 
1980).  Similar  programs  were  initiated  by  several 
physician-researchers  during  this  period  (Brown 
and  Uhl,  1970;  Fleisher,  Brown,  Zeleznik,  and 
others,  1976;  Williamson,  Aronovitch,  Simonson, 
and  others,  1975)  and  followed  the  same  form  as 
the  famous  Shewhart  cycle  (Berwick,  1991). 

The  second  form  of  mandatory  QA  was  Federal 
legislation  setting  up  the  Professional  Standards 
Review  Organization  (PSRO)  program  for  the 
Medicare  and  Medicaid  programs.  The  Federal 
Health  Care  Financing  Administration  (HCFA) 
contracted  with  PSROs,  which  were  voluntarily 
formed  organizations  of  physicians  practicing  in 
defined  areas.  The  program  evolved  slowly,  but  at  its 


peak  involved  three  types  of  review:  (1)  utilization 
review,  or  case-by-case  review  of  necessity  of  hospi- 
tal admission  and  length  of  stay;  (2)  profile  analysis, 
an  analysis  of  hospital  claims  data  bases  to  monitor 
patterns  of  practice  concerning  admission  and  length 
of  stay;  and  (3)  medical  care  evaluation,  an  activity 
analogous  to  the  JCAH's  patient  care  evaluation 
(Goran,  Roberts,  Kellogg,  and  others,  1975). 

Although  physician-experts  promoted  medical 
care  evaluation  as  a  means  to  improve  quality 
(Sanazaro  and  Worth,  1978),  this  did  not  become  a 
priority  for  HCFA.  Congress  and  the  Executive 
Branch  were  interested  in  QA  as  a  means  to  control 
costs  by  reducing  unnecessary  procedures  and  days 
of  hospital  care  (Health  Care  Financing  Adminis- 
tration, 1980).  As  a  result,  the  PSRO  program 
became  focused  primarily  on  utilization  review. 

Both  internal  and  external  quality  review  were 
resisted  by  many  physicians,  who  saw  them  as  an 
attack  on  professional  autonomy.  External  utiliza- 
tion review  was  particularly  unpopular  because 
review  was  no  longer  conducted  by  known  peers. 
Even  physicians  who  conceded  a  role  for  external 
review  often  argued  that  government  should  not 
concern  itself  with  the  mainstream  of  competent 
physicians  but  should  limit  itself  to  catching  "outlier 
physicians"  (Colcock,  1974;  Haggerty,  1974). 

Professional  resistance  to  QA  was  bolstered 
because  QA,  when  implemented  within  bureau- 
cratic institutions,  had  little  impact  on  quality.  QA 
committees  were  unable  to  change  institutional 
systems  for  implementing  care  because  the  impetus 
for  change  was  thwarted  by  the  inertia  inherent  in 
hospital  hierarchies.  As  JCAH  reports  of  the  time 
show,  many  institutions  did  not  want  to  change  the 
way  they  worked;  they  preferred  to  resist  passively 
by  interpreting  QA  as  compliance  with  JCAH 
requirements  "on  paper"  rather  than  in  reality 
(Roberts,  Coale,  and  Redman,  1987). 

New  Industrial  Model — Health  Care  as 
Corporate  Enterprise 

During  the  1980s,  public  policy  emphasized 
introducing  competition  in  health  care  as  a  way  to 
force  providers  to  behave  more  like  managers  of 
for-profit  corporations.  The  key  distinction  between 
for-profit  corporations  and  bureaucracies  is  that 
the  former  sell  their  services  in  a  competitive  mar- 
ket whereas  bureaucracies  do  not  (Downs,  1967). 
The  choices  made  by  informed  consumers  influence 
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the  profits  and  market  share  of  a  corporation  and 
so  determine  corporate  survival.  For-profit  corpora- 
tions, particularly  those  concerned  with  complex 
technology  and  rapidly  changing  product  lines, 
behave  differently  from  bureaucracies  because  they 
need  flexibility  to  survive  in  a  competitive  environ- 
ment (Lawrence  and  Lorsch,  1967).  Corporate  direc- 
tors delegate  control  of  work  to  managers,  who  are 
charged  to  produce  by  whatever  means  they  deem 
best.  Managers  have  authority  to  change  the  produc- 
tion system.  Product  planning  is  done  informally  in 
interdisciplinary  groups  that  operate  by  cooperation 
and  coordination.  Job  performance  is  judged  not  by 
conformance  to  rule  but  by  externally  valued  out- 
puts— that  is,  profitability  and  market  share. 

In  the  industrial-model  health  care  system  the 
two  major  classes  of  participants  adopt  new  roles. 
Patients  become  customers  or  consumers  rather 
than  beneficiaries.  Physicians  either  are  employed 
by  provider  organizations  or  contract  with  them  as 
business  partners.  Provider  organizations  include 
hospitals  and  HMOs,  as  well  as  new  types  of  organi- 
zations such  as  large  vertically  integrated  health 
care  plans  and  managed-care  organizations.  These 
organizations  employ  professional  and  technical 
personnel  and  an  increasing  number  of  managers 
who  expand  their  activities  into  marketing  the  orga- 
nization's services.  Few  physicians  are  now  in  solo 
practice  who  are  paid  directly  by  their  patients. 

How  does  the  industrial  model  work?  In  the 
form  of  the  industrial  model  as  advocated  by 
Enthoven  (1980),  customers  choose  among  health 
care  plans;  they  do  not  choose  individual  physicians. 
In  the  1990s,  this  pattern  became  reality  for  individ- 
uals purchasing  health  care,  since  physicians  tend 
to  work  with  health  care  plans  as  employees  or 
contractors.  Health  care  plans'  decisions  to  contract 
with,  or  to  hire,  retain,  and  promote  individual 
physicians  may  be  based  on  their  contribution  to 
the  productivity  of  the  organization.  A  health  care 
plan  also  may  negotiate  with  physicians  to  develop 
practice  guidelines,  whose  goal  is  to  ensure  consis- 
tency of  the  health  care  provided  by  the  plan 
(Gottlieb,  Margolis,  and  Schoenbaum,  1990). 

There  is  legitimate  concern  that  the  commercial 
incentives  inherent  in  a  competitive  health  care 
system  create  a  conflict  of  interest  for  the  physician. 
Most  problematic  are  situations  where  physicians 
can  profit  by  referring  patients  to  diagnostic  facili- 
ties or  by  prescribing  therapeutic  modalities  that 
they  own  or  in  which  they  are  invested.  Federal 


regulations  are  expected  in  the  near  future  to  limit 
physicians'  opportunities  to  promote  use  of  those 
services  in  which  they  have  a  direct  financial  interest 
(American  Medical  Association,  1992).  Conflict  of 
interest  is  also  present  in  some  form  in  the  other 
two  models  discussed  here:  in  the  fee-for-service 
system  traditionally  associated  with  the  professional 
model,  physicians  can  profit  by  prescribing  their 
own  services;  and  in  a  bureaucratic  model  where 
physicians  are  on  salary,  the  opportunity  for  physi- 
cians to  underprovide  their  own  time  and  effort 
presents  a  conflict  of  interest. 

Payment  in  the  industrial  model.  Payers  in  the 
industrial  model  include  self-insured  business 
corporations  acting  on  behalf  of  their  employees, 
as  well  as  the  Federal  and  State  governments  acting 
for  Medicare  and  Medicaid  enrollees.  If  these 
purchasers  buy  directly  from  individual  providers 
in  fee-for-service  arrangements,  they  generally  use 
fiscal  intermediaries  or  external  review  agencies  to 
control  costs  by  managing  utilization  of  services. 
Alternatively,  payers  now  take  active  roles  as 
"prudent  purchasers":  instead  of  setting  premiums 
to  cover  bills  approved  by  physicians,  they  contract 
for  services  with  health  care  plans,  negotiating  a 
fixed  price  per  enrollee.  Corporations,  especially, 
often  make  purchasing  agreements  with  several 
health  care  plans  to  offer  a  controlled  choice  of 
plans  to  their  employees. 

Health  care  plans  both  manage  the  insurance 
function  and  provide  care  either  by  employing  or 
contracting  with  physician  organizations.  Health 
care  plans  with  fixed-price  contracts  are  at  financial 
risk  for  whatever  care  their  patients  may  need.  This 
element  of  risk  provides  an  incentive  to  control 
costs  through  efficiency  in  clinical  and  administra- 
tive management.  However,  health  care  plans  may 
choose  to  control  costs  in  ways  that  purchasers  find 
undesirable,  such  as  discouraging  sicker  patients 
from  enrolling,  limiting  access  to  services,  and 
skimping  on  quality.  Accordingly,  purchasers  are 
increasingly  interested  in  quality  measurements  that 
would  enable  them  to  select  health  care  plans  and 
negotiate  contracts  on  quality  as  well  as  price. 

Quality  in  the  industrial  model.  In  the  industrial 
model  the  stimulus  for  quality  comes  from  the  need 
to  survive  in  a  competitive  environment.  Customer 
preferences  determine  the  market  share  of  each 
health  care  plan;  market  share  and  productivity 
determine  the  success  of  the  provider  organization. 
The  effectiveness  of  such  market  mechanisms 
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depends  upon  the  ability  of  customers  to  distin- 
guish differences  in  quality.  Consumers  have  lim- 
ited ability  to  judge  the  quality  of  health  care,  but 
recent  advances  in  quality  measurement  show 
promise  of  changing  this  situation.  A  variety  of 
valid  measures  of  quality  are  becoming  available. 
As  they  are  used  by  a  greater  number  of  parties, 
the  cost  of  measurement  becomes  more  support- 
able. These  sophisticated  measures  of  quality  are 
difficult  for  the  individual  consumer  to  interpret. 
Large  purchasers,  however,  can  employ  specialists 
who  are  understand  measures  of  quality.  In  this 
way,  governments  and  corporations  are  increasingly 
able  to  shop  for  value  by  making  informed  choices 
on  quality  as  well  as  cost. 

To  facilitate  competition  on  quality,  some  new 
roles  are  emerging  for  government  and  other  exter- 
nal reviewers.  For  instance,  the  Agency  for  Health 
Care  Policy  and  Research  (AHCPR),  by  facilitating 
development  of  practice  guidelines,  can  be  seen  as 
setting  health  care  industry  standards,  somewhat 
like  a  traditional  bureau  of  standards.  Some  State 
governments,  such  as  Pennsylvania's,  are  taking  an 
active  role  as  consumer  guides.  They  collect  infor- 
mation on  quality  and  provide  it  to  consumers  and 
bulk  purchasers  to  help  them  make  choices  (Office 
of  Technology  Assessment,  1988).  The  JCAHO 
"clinical  indicators"  program,  launched  in  1986,  will 
collect  data  from  hospitals  and  feedback  perfor- 
mance measures  showing  comparisons  of  quality 
of  care  among  hospitals  of  a  similar  type.  This  effort 
is  intended  to  facilitate  QI  by  providing  "bench- 
marks" that  reveal  the  best  performance  currently 
achievable  (Joint  Commission  on  Accreditation  of 
Healthcare  Organizations,  1990).  HCFA,  in  its 
"Health  Care  Quality  Improvement  Initiative," 
(HCFA)  also  plans  benchmarking  activities  (Jencks 
and  Wilensky,  1992). 


Emergence  of  Continuous  Quality 
Improvement  and  Total  Quality 
Management 

During  the  1980s,  American  corporations  began 
to  revise  and  expand  their  approach  to  quality.  QA 
activities  were  absorbed  into  a  more  comprehensive 
strategy  for  quality  management.  Garvin  (1988)  dis- 
tinguished the  characteristics  of  "strategic  quality 
management"  as  follows:  quality  is  a  constant  pre- 
occupation for  top  leadership;  quality  is  linked  in 


managers'  minds  to  profitability;  quality  is  defined 
from  the  customer's  point  of  view;  corporations 
compete  on  quality;  CQI  is  seen  as  a  way  to  exceed 
competitors,  not  just  as  a  means  to  meet  rules  or 
standards;  quality  management  pervades  all  work 
processes. 

This  new  approach  to  quality  includes  the  con- 
cepts of  CQI  and  TQM.  It  has  produced  startling 
improvements  in  quality  in  several  large  American 
corporations  that  manufacture  complex  technical 
products,  for  example,  Motorola,  Xerox,  Ford,  and 
the  Bridgestone  Tire  Company  (Berwick,  Godfrey, 
and  Roessner,  1990).  Since  health  care  involves 
much  complex  technology,  it  is  not  surprising  that 
CQI  and  TQM  are  now  being  applied  to  health 
care.  The  evolution  from  QA  to  CQI/TQM  is  con- 
sonant with  and  facilitated  by  the  emergence  of  the 
industrial  model  for  production  and  financing  of 
health  care. 

Quality  measurement  is  an  essential  feature  of 
the  industrial  model  for  health  care.  I  will  review 
the  advances  in  quality  measurement  gained  during 
the  era  of  QA  and  how  the  best  of  that  experience 
can  be  carried  forward  into  the  new  era  of  QI. 

Quality  Assurance 

Since  the  1960s,  QA  research  has  produced 
substantial  advances  in  quality  measurement.  We 
have  advanced  from  identifying  single  instances  of 
poor  quality  through  case-by-case  review  to  quality 
measurement  for  groups  of  cases  using  computer- 
ized review  systems.  Leaders  among  provider 
organizations  use  internal  quality  measures  to  main- 
tain standards  of  quality.  External  regulators  have 
begun  to  use  quality  measures  to  monitor  providers 
in  an  effort  to  guarantee  to  the  public  that  standards 
are  maintained. 

Dimensions  of  quality  of  care.  Quality  measure- 
ment estimates  the  magnitude  of  a  desired  attribute. 
In  broad  terms,  the  attribute  we  call  quality  of  care 
is  the  propensity  to  benefit  patients.  Providing  care 
that  benefits  patients  is  a  complex  enterprise  requir- 
ing many  different  components  and  activities.  We 
extrapolate  our  definition  of  quality  to  these  subor- 
dinate activities  by  specifying  the  attributes  we 
desire  for  them,  known  as  their  "key  quality  charac- 
teristics." Three  major  categories  of  key  quality 
characteristics,  sometimes  called  dimensions  of 
quality,  can  be  identified  (Palmer,  1991): 
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1.  Technical  quality  is  the  degree  to  which 
providers  coordinate  judgment,  skill,  and  avail- 
able technology  to  improve  the  health  of 
patients  (Palmer,  1991).  This  dimension  includes 
both  the  choice  of  an  appropriate  item  of  care 
and  its  safe  and  timely  performance.  For  invasive 
procedures,  Brook  (1989)  defines  appropriate 
care  by  posing  the  question,  "Does  the  health 
benefit  of  a  procedure  exceed  its  risk  by  a 
sufficiently  wide  margin  to  make  the  procedure 
worth  doing?"  The  two  attributes  "appropriate- 
ness" and  "good  performance"  are  captured  in 
the  phrase  "doing  the  right  things  and  doing 
them  right  the  first  time." 

2.  Accessibility  is  the  ease  with  which  health  care 
can  be  reached  by  a  beneficiary  in  the  face  of 
financial,  geographical,  organizational,  cultural, 
and  emotional  barriers  (Palmer,  1991). 

3.  Acceptability  is  the  degree  to  which  health  care 
satisfies  patients  (Palmer,  1991). 

Definitions  are  also  needed  for  three  related 
concepts  that  are  related  to  quality  of  care:  efficacy, 
effectiveness,  and  efficiency.  Efficacy  and  effective- 
ness relate  to  the  appropriateness  dimension  of 
quality  (Palmer,  1976, 1991).  To  measure  appropri- 
ateness, we  test  whether  providers  "did  the  right 
thing" — that  is,  whether  they  used  medical  tech- 
nologies such  as  tests,  procedures,  and  treatments 
correctly.  This  determination  of  appropriate  use 
presumes  knowledge  about  medical  technologies 
and  how  well  they  work.  Acquiring  this  knowledge 
is  the  task  of  technology  assessment.  Efficacy  and 
effectiveness  are  terms  used  in  assessing  technolo- 
gies. Efficacy  is  the  probability  of  benefit  to  individ- 
uals in  a  defined  population  from  a  medical  technol- 
ogy applied  for  a  given  medical  problem  under  ideal 
conditions  of  use.  Efficacy  is  measured  under 
strictly  controlled  conditions,  such  as  a  randomized 
controlled  trial.  Effectiveness  is  the  probability  of 
benefit  to  individuals  in  a  defined  population  from  a 
medical  technology  applied  for  a  given  medical 
problem  under  average  conditions  of  use.  So  the 
effectiveness  of  a  technology  depends  not  only  on 
the  efficacy  of  the  technology  but  also  on  the  techni- 
cal quality  of  the  performance  of  the  average 
provider,  because  the  probability  of  benefit  to  the 
patient  will  be  diminished  if  the  average  provider 
chooses  the  technology  inappropriately  or  imple- 
ments it  poorly. 


Efficiency  is  a  concept  related  to  both  cost  and 
quality  of  care.  It  is  defined  by  the  degree  of  quality 
present  in  a  given  unit  of  health  care  for  which  a 
fixed  amount  of  resources  is  used  (Palmer,  1991). 
More  quality  bought  for  the  same  amount  of 
resource  use  indicates  that  the  care  was  more 
efficiently  produced.  We  also  can  invert  the  concept 
and  define  efficiency  in  terms  of  the  amount  of 
resources  used  to  produce  a  given  unit  of  health 
care  of  a  fixed  degree  of  quality.  Fewer  resources 
used  to  create  the  same  amount  of  quality  indicates 
greater  efficiency.  In  either  case,  greater  efficiency 
means  more  value  for  money. 

Quality  measurement.  A  quality  attribute  cannot 
be  observed  directly  because  it  is  an  abstract  notion. 
For  actual  measurements,  we  observe  quality 
attributes  as  revealed  in  samples  of  care  given  to 
patients.  We  infer  presence  of  the  attribute  by 
detecting  an  event  that  would  occur  if  the  attribute 
were  present.  The  instances  of  patient  care  that  we 
observe  become  the  sample  of  care  used  for 
measurement.  For  instance,  we  observe  whether 
recovery  (a  desired  event)  occurred  for  a  patient 
who  received  a  coronary  artery  by-pass  graft;  if 
recovery  occurred,  we  assume  that  the  patient 
received  good  quality  of  care.  All  patients  who 
experienced  coronary  artery  by-pass  grafts  make  up 
the  sample  used  for  measurement.  We  also  can 
count  undesired  events — such  as  death — as  evidence 
that  the  patient  received  poor  quality  of  care;  these 
are  sometimes  called  adverse  or  sentinel  events. 

Observing  a  single  indicator  event  does  not  per- 
mit measurement  but  only  an  isolated  quality  judg- 
ment. Measurement  is  an  estimate  of  magnitude,  a 
question  of  "how  much?"  To  measure  a  quality 
attribute,  we  borrow  methods  from  epidemiology 
and  compute  rates  of  occurrence  of  desired  or  unde- 
sired events.  The  JCAHO  calls  these  "rate-based" 
indicators  (Joint  Commission  on  Accreditation  of 
Healthcare  Organizations,  1990).  For  instance,  we 
count  all  relevant  instances  of  care  experienced  by 
patients  in  the  chosen  sample  in  the  denominator  of 
the  rate;  these  are  the  denominator  events.  For  a 
positive  indicator,  we  count  all  instances  in  which 
those  patients  experienced  desired  events  in  the 
numerator  of  the  rate;  these  are  the  numerator 
events.  The  criteria  set  to  define  the  numerator 
events  become  the  units  of  measurement.  One  way 
for  an  HMO  to  estimate  its  success  in  providing  pre- 
natal care  is  to  calculate  the  proportion  of  its 
enrollees  who  received  satisfactory  prenatal  care 
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(the  numerator  events)  out  of  all  those  whose 
pregnancies  began  within  a  1-year  period  (the 
denominator  events).  The  unit  of  measurement  is  1- 
year's-worth  of  pregnancy  episodes. 

Measurement  error.  Measurement  provides  an 
estimate  of  an  attribute  of  quality,  but  estimates 
always  have  some  error.  When  we  observe  variation 
in  a  series  of  measurements,  we  interpret  the  differ- 
ences between  measurements  to  mean  that  real  dif- 
ferences in  quality  exist.  If  our  measure  were  error- 
free,  this  would  be  so.  However,  there  may  be 
sources  other  than  true  variations  in  the  quality  of 
care  that  account  for  the  variation  in  the  series  of 
measurements;  these  are  sources  of  measurement 
error  (McAuliffe,  1979). 

Error  may  be  either  systematic  or  random.  If  a 
measurement  is  reliable,  the  rate  of  random  error  is 
low,  but  there  still  may  be  systematic  error — that  is, 
we  are  measuring  some  attribute  but  not  the  one 
we  want  to  measure.  If  a  measure  is  valid,  both 
random  and  systematic  error  are  low.  When  we 
know  that  proposed  denominator  and  numerator 
events  are  influenced  by  sources  of  variation  other 
than  the  intended  quality  attribute,  we  try  to 
exclude  the  sources  of  error  by  respecifying  the 
measure.  Alternatively,  we  retrospectively  adjust 
the  measurements  we  obtain  to  correct  for  the 
sources  of  error.  In  addition,  we  can  use  statistical 
models  to  predict  the  expected  occurrence  of 
desired  (or  undesired)  events,  while  adjusting  the 
analysis  for  components  of  variance  related  to 
factors  other  than  differences  in  quality. 

Major  sources  of  error  are  inaccuracies  in 
recording,  coding,  collecting,  and  transmitting  data 
about  patient  care  events.  Other  sources  of  error  lie 
in  the  interpretation  of  patient  care  events.  Most 
events  have  multiple  causes;  therefore,  to  measure 
the  contribution  of  any  single  cause  one  must  adjust 
for  the  contribution  of  others  (Orav,  Louis,  Palmer, 
and  Wright,  1991).  It  is  easiest  to  measure  perform- 
ance of  a  whole  system,  which  includes  contribu- 
tions of  all  individuals  within  it;  the  main  source  of 
error  then  is  variation  contributed  by  patients 
whose  care  is  studied  and  the  circumstances  of  time 
and  place  when  the  measurement  was  made.  If  we 
want  to  measure  performance  of  a  group  of  physi- 
cians, then  it  is  a  source  of  error  to  include  events 
that  are  controlled  even  in  part  by  the  system,  by 
patients,  or  by  time  and  place  when  care  was  given. 
Inevitably  it  is  more  difficult  to  measure  perform- 
ance for  the  individual  physician  than  for  the 


group;  sources  of  error  in  measuring  performance 
of  the  individual  physician  include  contributions  to 
the  error  rate  by  patients,  the  system,  and  other 
physicians  to  the  extent  that  they  control  the 
indicator  events. 

Measurement  of  performance  of  individual 
physicians  is  essential  to  one  strategy  for  QA:  the 
"outlier"  approach.  This  requires  finding  providers 
who  are  performing  much  worse  than  others,  the 
outliers.  The  motive  is  to  either  change  their 
performance  or  exclude  them  from  giving  care.  In 
looking  for  outliers,  the  reviewer  wants  to  know 
"did  this  provider  experience  more  undesired 
events  (or  fewer  desired  events)  than  would  be 
expected  in  a  group  of  similar  providers  if  he  or 
she  was  in  the  same  group  as  all  providers?" 
(Kritchevsky  and  Simmons,  1991;  Orav,  Louis, 
Palmer,  and  Wright,  1991).  It  is  inherently  difficult 
to  make  measurements  at  this  level.  With  individual 
physicians,  numbers  of  observations  are  harder  to 
accumulate  than  for  a  group  of  physicians,  and  small 
numbers  make  estimates  less  reliable.  A  more  com- 
plex data  base  is  required  to  measure  performance 
at  the  individual-physician  level — for  example, 
one  must  attribute  each  of  the  denominator  and 
numerator  events  to  an  individual  provider,  and  all 
other  sources  of  error  must  also  be  linked  to  the 
individual  provider. 

Defining  denominator  events.  As  we  have 
discussed,  quality  measurements  are  rates  with  a 
numerator  and  a  denominator.  Defining  eligibility 
for  the  denominator  group  means  specifying  the 
sample  of  patient  care  events  to  be  observed. 
Within  that  sample,  all  numerator  events  must  be 
observed  and  registered.  A  sample  for  quality 
measurement  must  be  specified  as  to  the  type  of 
data  used,  the  data  source,  the  method  for  finding 
(indexing)  the  sample  of  cases,  the  period  of  enroll- 
ment of  cases  in  the  sample,  the  time  window  for 
data  collection,  the  methods  and  specific  instructions 
for  data  collection,  and  the  unit  of  care  to  which  the 
quality  assessment  is  applied. 

1.  Data  type.  This  refers  to  Donabedian's  classifica- 
tion into  structure  (data  about  facilities,  equip- 
ment, supplies,  policies,  and  procedures); 
process  (data  about  things  done  to  or  for  or  by 
patients);  and  outcome  (those  changes  in 
patients'  health  status  that  are  attributable  to  the 
care  received).  Data  about  structure  are  mostly 
used  when  provider  organizations  are  the  object 
of  study.  However,  having  a  good  structure  for  a 
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hospital  or  HMO  does  not  reveal  anything  about 
the  actual  care  given,  only  about  the  capacity  to 
produce  good  care. 

2.  Data  sources.  Commonly  used  sources  include 
data  compiled  by  providers  such  as  medical 
records  and  claims  data,  data  contributed  by 
patients  in  response  to  interviews  and  question- 
naires, and  data  acquired  in  the  course  of  a 
quality  judgment  as  in  surgical  second  opinions 
and  in  preprocedure  reviews. 

3.  The  index  for  finding  the  sample  of  cases.  Cases 
are  commonly  sampled  by  diagnosis  from  a 
diagnostic  index  or  from  data  tapes  of  claims 
that  record  a  diagnosis  for  a  visit  or  hospitaliza- 
tion. Sampling  the  correct  denominator  events 
for  a  measurement  is  a  key  step  in  creating  a 
valid  measurement;  when  diagnosis  defines  the 
denominator  events,  thoughtful  specification  of 
the  diagnosis  limits  measurement  error. 

4.  The  period  of  enrollment  of  the  sample.  Cases 
are  often  sampled  from  a  year's  worth  of  data 
to  allow  for  seasonal  variation  in  the  types  of 
patients  and  their  illnesses  and  in  staffing 
arrangements. 

5.  The  time  window  for  data  collection.  This  is  the 
period  for  which  care  is  observed  for  each  case 
in  the  sample.  It  may  be  defined  as  a  fixed 
period  of  time,  an  episode  of  hospitalization, 
or  an  episode  of  illness.  Episodes  have  appeal 
as  a  natural  unit  of  clinical  work,  but  they  are 
difficult  to  use  because  of  problems  in  defining 
the  beginning  and  end  of  an  episode  and  varia- 
tions in  the  time  interval  from  case  to  case  in 
the  sample. 

6.  The  methods  and  specific  instructions  for  data 
collection.  Uncontrolled  variation  in  procedures 
for  data  collection  is  a  major  source  of  unrelia- 
bility in  quality  measurements.  Rigorous  quality 
control  of  data  at  every  step  in  recording,  coding, 
and  processing  is  critical  for  limiting  measure- 
ment error. 

7.  The  unit  of  care  to  which  the  quality  assessment 
is  applied.  Most  commonly,  a  judgment  is  made 
for  each  case  in  a  sample.  These  case-specific 
judgments  may  then  be  aggregated  to  create  a 
profile  of  performance  for  an  individual  practi- 
tioner or  for  a  provider  institution.  Profiles  for 
many  physicians  or  hospitals  can  be  aggregated 


to  describe  a  pattern  of  care.  In  other  instances, 

data  from  many  cases  for  the  same  physician  or 

hospital  are  pooled,  and  the  quality  judgment  is 

applied  to  the  pooled  data. 

Note  that  each  of  these  data  characteristics 
potentially  contributes  error  to  a  quality  measure- 
ment. It  is  easier  to  control  for  sources  of  error 
when  more  specific  details  attaching  to  case, 
provider  of  care,  practice  setting,  and  time  and 
place  of  care  are  recorded  along  with  the  details  of 
care  given.  The  current  trend,  therefore,  is  to  use 
computerized  data  bases  for  quality  measurement 
that  permit  speedy  analysis  of  a  detailed  matrix  of 
information  about  patient  care. 

Defining  numerator  events.  Once  a  sample  of 
denominator  events  has  been  defined,  we  can 
address  how  to  judge  which  of  these  events  consti- 
tute good  or  bad  care.  The  means  for  making  this 
judgment  is  a  criterion.  A  criterion  specifies  how  to 
classify  a  denominator  event  in  terms  of  the  chosen 
key  quality  characteristic  and  thus  decide  whether 
to  count  the  event  in  the  numerator.  Usually,  a  cate- 
gorical judgment  (good/bad)  is  made,  although 
occasionally  a  finer  gradation  is  used,  such  as  inade- 
quate, minimal,  good,  and  excellent. 

Q A  initially  was  done  by  peer  review  of  individ- 
ual cases.  A  physician  who  was  trusted  to  be  an 
arbiter  of  quality  reviewed  the  data  for  each  case 
and  used  his  or  her  own  judgment  to  classify  the 
case.  This  is  called  using  "implicit"  judgment,  or 
"implicit  criteria,"  because  the  criteria  on  which  the 
judgment  is  based  remain  concealed  in  the  mind  of 
the  judge.  Not  surprisingly,  as  with  any  form  of 
human  judgment,  implicit  judgments  are  found  to 
be  unreliable;  the  same  unit  of  care  may  be  judged 
differently  by  different  judges  and  even  by  the  same 
judge  on  different  occasions.  One  can  improve  the 
reliability  of  implicit  judgments  by  ensuring  that  all 
judges  use  the  same  data  set,  by  using  more  than 
one  judge  and  aggregating  judges'  findings,  and  by 
directing  judges  to  focus  on  particular  quality  char- 
acteristics. This  last  maneuver  is  called  "structuring 
peer  review." 

By  asking  judges  to  explain  the  basis  of  their 
judgments,  we  arrive  at  the  idea  of  "explicit  criteria," 
in  which  the  reason  for  classifying  care  as  good  or 
bad  is  defined  in  writing  before  review  begins. 
Explicit  criteria  sets  have  become  more  sophisti- 
cated by  the  incorporation  of  techniques  used  in 
medical  decision  analysis;  they  now  consist  of  algo- 
rithms with  branching  logic  that  accommodate  a 
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variety  of  patient  conditions.  If  the  data  on  denomi- 
nator events  and  the  data  needed  to  classify  events 
as  numerator  events  are  in  a  computerized  data 
base,  the  algorithm  can  be  programmed  to  compute 
the  indicator  rate.  The  set  of  measurements  resulting 
from  this  program  can  be  compiled  by  a  scoring  pro- 
gram to  yield  a  set  of  performance  measurements. 

Criteria  may  refer  to  a  broad  range  of  denomi- 
nator events,  such  as  all  hospitalized  patients.  In  this 
case,  they  are  called  "generic."  When  criteria  refer 
to  a  single  diagnostic  category,  procedure,  health 
state,  drug,  or  test,  they  are  called  specific.  Criteria 
may  be  developed  by  deciding  what  should  be  done, 
what  should  be  the  outcome  (prescriptive  criteria), 
or  by  comparing  what  is  done  or  which  outcomes 
occur  among  different  providers  (comparative  crite- 
ria). Comparative  criteria  are  most  often  used  with 
pooled  outcome  data,  for  which  it  is  difficult  to 
define  what  should  be  deemed  good  quality. 

Quality  measurements  usually  take  the  form  of 
rates,  a  continuous  variable.  For  any  rate,  it  may  be 
desirable  to  set  some  cutoff  point — a  standard 
— that  determines  whether  action  should  be  taken 
based  on  this  finding.  Standards  themselves  may  be 
set  prescriptively  or  comparatively.  Minimal  stand- 
ards set  a  floor;  if  care  falls  below  this  level,  some 
sanction  or  corrective  action  is  required  (Field  and 
Lohr,  1990).  Alternatively,  a  standard  may  be  a 
benchmark  to  show  the  best  performance  achieved; 
this  is  used  to  spur  continuous  efforts  to  reach  the 
level  of  performance  at  the  benchmark  (Field  and 
Lohr,  1990). 

A  different  type  of  standard  is  used  for  attrib- 
utes that  should  remain  within  a  given  range  so  long 
as  a  procedure  is  under  control.  Standards  set  as  a 
range  function  like  a  traditional  industrial  quality- 
control  chart;  care  falling  outside  the  range,  care 
that  is  highly  variable  within  the  range,  or  care  that 
is  drifting  directionally  within  the  range  may  require 
investigation  and  action.  This  approach  could  be 
used  for  issues  of  over-  and  underutilization. 

When  explicit  criteria  are  used,  they  cannot 
branch  sufficiently  to  cover  all  possible  patient  cir- 
cumstances that  might  affect  the  process  or  outcome 
of  care.  To  branch  for  all  circumstances,  including 
rare  ones,  would  make  a  measure  too  cumbersome 
and  expensive  to  develop  and  use.  A  common 
practice  is  to  use  explicit  review  and  then  arrange 
implicit  peer  review  of  cases  that  fail  to  meet  criteria. 
The  peer  reviewer  judges  whether  this  case  has 


some  rare  circumstance,  which  would  mean  that  the 
criterion  could  not  or  should  not  be  met. 

Peer  review  has  been  used  by  the  Peer  Review 
Organization  (PRO)  program  since  the  1980s  for 
case-by-case  review  of  each  instance  of  care  that 
fails  generic  screening  criteria  designed  to  detect 
adverse  patient  care  events.  The  peer  reviewer 
decides  whether  corrective  action  should  be  taken 
for  the  physician  deemed  responsible  for  the 
adverse  event.  In  the  era  of  quality  measurement, 
peer  review  of  a  sample  of  cases  can  be  used  to  test 
a  performance-measuring  tool  for  its  inherent  error. 
Although  measurement  error  cannot  be  eliminated 
altogether  without  making  the  measurement  instru- 
ment too  expensive  and  unwieldy,  knowing  the  mea- 
surement error  for  a  given  instrument  is  important, 
for  it  enables  us  to  trade  off  between  the 
accuracy  of  measurement  that  we  want  and  the 
amount  we  can  spend  on  measurement. 

Process  measures:  Practice  guidelines.  Measures 
of  technical  quality  that  are  based  on  process  data 
deal  with  what  providers  should  do.  But  how  do 
providers  know  what  they  should  do?  Earlier  we 
quoted  a  definition  for  appropriateness  of  care 
indicating  that  they  should  do  what  will  benefit 
their  patients.  Determining  what  will  benefit  patients 
is  the  mission  of  clinical  research.  However,  the 
traditional  method  for  testing  the  benefits  to  be 
gained  from  medical  technologies  is  the  randomized 
controlled  trial,  which  gives  evidence  only  about 
the  benefit  to  be  expected  from  a  technology  when 
used  under  ideal  conditions.  Considerable  extrap- 
olation is  needed  to  derive  what  should  be  done  in 
all  other  conditions. 

To  address  this  problem,  AHCPR  now  funds 
Patient  Outcomes  Research  Teams  (PORTs)  to 
analyze  administrative  data  bases  and  derive  infor- 
mation about  effectiveness  of  medical  technologies 
— that  is,  the  likelihood  that  they  will  benefit 
patients  under  average  conditions  of  use.  Informa- 
tion from  this  source  and  from  traditional  clinical 
research  can  be  used  to  formulate  practice  guide- 
lines for  providers. 

Practice  guidelines  are  defined  by  the  Institute 
of  Medicine  as  "systematically  developed  state- 
ments to  assist  practitioner  and  patient  decisions 
about  appropriate  care  for  specific  clinical  circum- 
stances" (Field  and  Lohr,  1990).  Currently,  there  is 
a  major  effort  within  the  medical  profession  to 
develop  guidelines  for  what  providers  should  do 
in  particular  situations.  AHCPR  is  mandated  to 
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facilitate  development  of  such  practice  guidelines 
(Omnibus  Budget  Reconciliation  Act,  1989).  When 
a  practice  guideline  (or  protocol,  or  clinical  policy) 
already  exists,  explicit  criteria  for  review  of  process 
data  can  be  based  on  the  practice  guideline.  When 
no  guideline  exists,  writing  process  review  criteria, 
in  effect,  defines  what  the  guideline  would  have 
been.  When  a  guideline  exists,  computerized 
reminders  can  prompt  providers  about  the  guideline 
when  action  is  needed  (Barnett,  Winickoff,  Dorsey, 
and  others,  1978;  McDonald,  1976),  and  computer- 
ized review  criteria  can  be  used  to  determine 
whether  guidelines  were  followed  (see  Table). 


Table.  Contrasting  guidelines  and 
review  criteria 


Practice  guidelines  Review  criteria 

(algorithms) 

Guide  care  to  be  given       Review  care  already 

given 

Data  obtained  as  required  Use  data  documenting 

care  given 

Cover  all  pathways  Cover  main  pathways 

Complement  clinical  Cover  main  pathways 
judgment 


When  one  is  looking  back,  trying  to  determine 
whether  the  guidelines  were  followed  by  a  provider, 
it  is  necessary  to  apply  review  criteria  to  data  about 
care  already  given  (Field  and  Lohr,  1990).  This  ret- 
rospective method  of  review  is  the  most  commonly 
used.  Prospective  judgment  of  quality  is  used,  how- 
ever, in  preprocedure  review  and  surgical  second 
opinion  review.  Here  a  judgment  of  appropriateness 
is  made  before  care  is  given,  so  inappropriate  proce- 
dures can  be  prevented. 

Outcome  measures.  As  large  corporations  have 
become  more  interested  in  purchasing  quality 
health  care,  it  has  become  common  to  draw  an 
analogy  between  the  notion  of  industrial  outputs 
and  health  outcomes.  A  simplified  diagram  of  an 
industrial  production  process  shows  inputs  (raw 
materials)  being  put  through  a  production  process 
that  results  in  outputs  (manufactured  products). 
The  idea  of  "management  by  outputs"  is  based  on 
this  model;  it  says  that  quality  is  improved  most  if 
managers  are  judged  by  their  outputs  rather  than  by 
conformance  to  work  specifications.  Too  often  this 


model  is  applied  to  health  care  in  a  simplistic  way 
that  implies  that  sick  patients  are  the  input  and 
healthy  patients  the  output.  This  analogy  is  not 
accurate  and  may  lead  to  undesirable  provider  and 
purchaser  behavior.  Manufacturers  control  outputs 
in  part  by  selecting  the  best  possible  raw  materials. 
If  this  thinking  is  applied  in  health  care,  it  condones 
providers  and  purchasers  who  select  less  sick 
patients  in  order  to  demonstrate  better  outcomes. 
Yet  excluding  sicker  patients  from  treatment  defeats 
the  purpose  of  health  care.  The  analogy  also  signifi- 
cantly overestimates  the  amount  of  control  that 
providers  have  over  disease.  Health  status  is  deter- 
mined by  many  factors  other  than  medical  care, 
including  genetic  makeup,  environmental  factors, 
the  number  and  severity  of  exposures  to  disease 
agents,  personal  lifestyle,  and  willingness  to  take 
action  in  behalf  of  one's  own  health.  Health  care 
providers  cannot  guarantee  improved  health  by 
following  specifications  for  their  work  in  the  way 
that  workers  following  specifications  can  determine 
the  outputs  of  a  production  process.  The  parallel 
with  industry  fits  better  if  we  see  health  services  as 
the  output  from  medical  providers;  the  quality  of 
these  services  can  be  measured  by  how  positively 
they  contribute  to  a  patient's  course  of  health.  Thus 
the  measure  of  quality  is  change  in  health  status 
attributable  to  the  health  care  given,  not  simply 
patient  health  status  after  receiving  care.  This  idea 
fits  with  the  definition  of  quality  advanced  by  the 
industrial  quality  expert  Juran  (1988),  namely 
"quality  is  fitness  for  use":  health  services  are  of 
good  quality  to  the  extent  that  they  are  fit  for  their 
intended  purpose  of  contributing  to  patients'  health. 

In  measuring  quality  by  use  of  outcome  data, 
we  must  control  for  all  risk  factors  influencing  a 
patient's  health.  This  approach  should  include 
adjustment  for  the  type  (case  mix)  and  severity  of 
the  patient's  illness(es),  any  comorbidities,  and 
relevant  environmental  and  psychosocial  variables 
(Greenfield,  Aronow,  Elashoff,  and  Watanabe, 
1988;  Jencks  and  Dobson,  1987;  Smith,  Pine,  Bailey, 
and  others,  1991).  Theoretically,  all  risk  factors 
should  be  measured  for  all  patients  at  the  start  of 
care  and  used  to  adjust  the  observed  outcome  data 
to  create  a  measure  of  quality  of  care.  Unfortunately, 
many  of  these  factors  are  themselves  hard  to  mea- 
sure, and  the  magnitude  of  their  contributions  to 
health  status  is  poorly  understood. 

During  the  1980s,  considerable  progress  was 
made  in  measuring  case  mix,  case  severity,  and 
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comorbidity  (Daley,  Jencks,  Draper,  and  others, 
1988;  Dubois,  Rogers,  Moxley,  and  Draper,  1987; 
Greenfield,  Aronow,  Elashoff,  and  Watanabe,  1988; 
Jencks  and  Dobson,  1987;  Smith,  Pine,  Bailey,  and 
others,  1991).  However,  most  case-mix  measures 
were  developed  for  adjustment  related  to  compar- 
isons of  resource  use  rather  than  for  outcome  pre- 
diction (Jencks  and  Dobson,  1987).  Those  measures 
tested  for  adjusting  patient  outcomes  were  designed 
for  use  in  predicting  mortality  and  are  not  necessar- 
ily suitable  for  other  types  of  outcomes  data. 
Adjusting  adequately  for  risk  factors  remains  an 
unsolved  problem. 

Outcomes  data  are  generally  used  with  compara- 
tive rather  than  prescriptive  criteria.  After  all  possi- 
ble adjustments  are  made  for  risk  factors  other  than 
quality  of  care  that  influence  the  outcome,  data  on 
outcomes  for  a  given  set  of  denominator  events  are 
compared  among  providers.  To  date,  death  and 
other  negative  outcomes  have  been  used  most  often 
to  measure  quality  of  care.  Now,  however,  there  are 
good  prospects  for  a  more  positive  approach  to 
quality  measurement  using  outcomes  data.  There 
have  been  great  advances  in  measuring  health  status 
in  terms  of  how  well  the  patient  is  feeling  and  func- 
tioning (Lohr  and  Ware,  1987).  Substantial  contri- 
butions were  made  by  two  large  studies,  the  RAND 
Health  Insurance  Experiment  (Newhouse, 
Manning,  Morris,  and  others,  1981)  and  the  Medical 
Outcomes  Study  (Tarlov,  Ware,  Greenfield,  and 
others,  1989).  The  latter  study  also  is  producing 
methods  for  measuring  health  and  function  for 
specific  diseases.  At  present,  however,  there  is  little 
work  on  case  adjustment  that  would  permit  these 
types  of  data  to  be  used  in  measuring  quality. 

The  newer  measures  of  patient  health  status  use 
data  collected  from  patients  by  questionnaire. 
Patients  also  can  provide  factual  data  about  the 
services  they  receive  (Davies  and  Ware,  1988).  This 
method  is  particularly  useful  if  the  providers  do  not 
keep  accurate  data  or  for  data  not  usually  recorded 
by  providers.  For  instance,  patients  can  be  asked 
how  long  it  took  to  get  medical  care;  by  applying  an 
objective  criterion  to  an  acceptable  waiting  time,  a 
measure  of  accessibility  of  care  is  created  (Mushlin 
and  Appel,  1980).  Through  questionnaires,  patients 
also  can  act  as  implicit  judges  of  acceptability  of  care 
by  rating  their  satisfaction  with  specific  characteris- 
tics of  care  (Davies,  Ware,  Brook,  and  others,  1986). 
These  ratings  are  analogous  to  what  industry  calls 
"measures  of  quality  from  the  users'  perspective." 


However,  the  patient  populations  served  by  differ- 
ent providers  may  not  be  uniform  in  the  level  of  ser- 
vice that  satisfies  them;  adjustments  for  patients' 
differing  levels  of  demand  are  therefore  needed 
when  providers  are  compared  for  their 
ability  to  satisfy  patients. 

How  can  measures  of  quality  be  evaluated? 
Quality  of  care  measures  can  be  evaluated  for  their 
reliability,  validity,  cost,  and  usefulness  for  a  desig- 
nated purpose  (McAuliffe,  1984).  I  have  already 
discussed  reliability  and  validity  of  measures  in 
terms  of  sources  of  random  and  systematic  error. 
Such  errors  can  occur  because  the  denominators 
and  numerators  specified  for  a  measurement  are 
inappropriate  or  because  of  errors  in  data  collection 
or  in  application  of  criteria  to  the  data.  Measures 
can  be  tested  for  validity  by  seeing  if  they  behave 
as  expected  in  comparison  with  other  measures 
applied  to  the  same  objects  of  measurement.  For 
instance,  we  can  test  whether  the  measures  of 
quality  being  evaluated  correlate  positively  with 
other  variables  believed  to  predict  good  quality  and 
correlate  negatively  with  variables  believed  to 
predict  poor  quality. 

Quality  assurance  in  the  1970s  and  1980s.  During 
the  1970s  and  1980s,  quality  assessments  often  were 
made  without  using  them  for  quality  measurement. 
Several  types  of  interventions  followed  directly 
upon  detection  of  even  a  single  highly  undesirable 
event.  For  instance,  in  the  first  three  rounds  of  PRO 
contracts,  a  PRO  that  confirmed  evidence  of  a  sin- 
gle gross  and  flagrant  violation  of  professional  stan- 
dards was  required  to  begin  a  QI  plan  for  the  physi- 
cian concerned.  Of  course,  the  physician  was 
guaranteed  due  process  to  dispute  the  evidence. 
Other  examples  of  action  for  a  single  violation 
include  mild  forms  of  sanction  such  as  denial  of  pay- 
ment if  preprocedure  review  suggests  that  a  surgical 
procedure  is  unnecessary.  Rutstein  suggested  moni- 
toring of  "sentinel  events" — truly  egregious  but 
rare  events  such  as  a  maternal  death — that  imply  a 
failure  in  quality  of  care  (Rutstein,  Berenberg, 
Chalmers,  and  others,  1976).  Any  such  event  would 
trigger  an  investigation  to  determine  what  had  gone 
wrong.  A  court  judgment  of  malpractice  also  leads 
to  a  punitive  intervention  based  on  a  presumed 
single  serious  failure  in  quality  of  care,  since  the 
judgment  implies  a  finding  that  an  injury  was  caused 
by  the  provider's  negligent  departure  from  the 
prevailing  standard  of  care. 
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A  problem  with  focusing  only  on  isolated  viola- 
tions is  that  a  great  deal  of  review  time  and  expense 
is  needed  to  uncover  a  single  case  that  warrants 
action.  It  is  an  unpleasant  activity  for  the  reviewers 
because  the  emphasis  is  on  assigning  blame. 
Another  disadvantage  is  that  goals  are  set  too  low 
because  this  method  implicitly  defines  quality  as 
the  absence  of  major  disasters.  In  industry  this  strat- 
egy— called  "quality  by  inspection" — is  regarded  as 
outmoded. 

In  addition  to  case-by-case  review,  quality  mea- 
surements were  in  widespread  use  as  early  as  the 
1970s,  when  both  the  JCAH  (Jacobs,  Christoffel, 
and  Dixon,  1976)  and  the  PSRO  program  required 
hospitals  to  conduct  internal  QA  (Goran,  Roberts, 
Kellog,  and  others,  1975).  A  QA  cycle  required 
quality  measurement,  taking  corrective  action  if  the 
findings  suggested  this  was  needed,  and  remeasure- 
ment  to  determine  whether  the  desired  results  had 
been  achieved.  The  requirements  were  obviously 
modeled  on  the  famous  "Shewhart"  cycle,  pro- 
moted in  industry  during  the  1930s  (Berwick,  1991). 

Four  obvious  problems  emerged  in  implement- 
ing these  requirements.  First,  measuring  quality 
proved  more  difficult  than  had  been  expected.  Since 
there  were  no  generally  accepted  practice  guide- 
lines, most  QA  committees  spent  their  time  getting 
agreement  on  review  criteria;  few  progressed  to  the 
QI  phase.  Data  bases  containing  sufficient  clinical 
detail  and  computers  sufficiently  powerful  to 
process  such  data  were  seldom  available  to  QA 
committees.  Also,  the  measurements  were  too 
global  and  did  not  deal  with  the  level  of  detail 
where  systems  changes  could  be  made  (Berwick, 
Godfrey,  and  Roessner,  1990). 

Second,  because  quality  measurement  was  so  dif- 
ficult and  expensive,  often  just  a  single  measurement 
was  made.  A  single  measurement,  made  with  invali- 
dated measures,  lacked  credibility  to  many  clinicians 
and  hospital  administrators  and  was  insufficient  to 
persuade  them  that  changes  might  be  needed. 

Third,  clinicians  and  administrators  in  hospitals 
thought  of  QA  in  bureaucratic  terms  as  a  method 
for  detecting  infractions  of  the  rules  and  blaming  the 
individual  responsible  for  the  infraction.  Few  QA 
committee  members  understood  the  concept  of 
system  change  as  a  more  powerful  way  to  improve 
quality  for  all  providers.  Even  if  they  had,  it  is  time 
consuming  and  tedious  to  make  changes  in  organi- 
zations composed  of  separate  hierarchies.  This 
problem  of  inertia  was  not  restricted  to  hospitals. 


A  Wall  Street  Journal  article  (White,  Patterson,  and 
Ingrassia,  1992)  identified  separate  hierarchies  in  the 
U.S.  automobile  industry  during  this  same  time 
period  as  a  barrier  to  producing  high-quality  vehicles. 
Last,  but  not  least,  most  physicians  resisted  involve- 
ment in  QA,  which  they  perceived  as  an  administra- 
tive activity  unrelated  to  their  own  work  goals. 

By  the  1980s,  PSROs  had  been  disbanded,  and 
the  JCAH  had  concluded  that  QA  requirements 
had  not  resulted  in  real  improvements  in  quality  in 
most  hospitals.  The  JCAH  therefore  reformulated 
its  QA  requirements,  emphasizing  repeated  mea- 
surement (monitoring),  responsibility  for  QA  within 
each  department  and  unit,  and  the  importance  of 
solving  problems,  rather  than  punishing,  as  the 
approach  to  QI  (Joint  Commission  on  Accredita- 
tion of  Hospitals,  1980). 

Quality  measurement  also  was  used  to  a  limited 
extent  during  the  1970s  and  1980s  as  a  means  for 
external  agencies  to  monitor  quality  of  care.  Just 
before  it  was  disbanded,  the  PSRO  program  had 
succeeded  in  getting  a  few  PSROs  to  analyze  pro- 
files of  use  of  hospital  bed  days  to  study  appropri- 
ateness of  use  (Goran,  1979).  By  the  time  PROs 
replaced  PSROs,  hospitals  had  incentives  to  reduce 
use  of  bed  days  because  they  were  paid  a  fixed  fee 
according  to  a  patient's  diagnosis-related  group 
(DRG)  classification  and  irrespective  of  the  num- 
ber of  days  of  stay.  The  resulting  rapid  decline  in 
lengths  of  stay  reveals  that  changing  hospitals' 
financial  incentives  was  an  effective  means  of 
changing  hospital  behavior  in  the  direction  desired 
by  policymakers. 

In  the  PRO  program  of  the  1980s,  profile  and 
pattern  analysis  reemerged  as  a  means  for  repeated 
measures  of  an  aspect  of  quality.  HCFA  has  used 
mortality  rates  adjusted  for  case  mix  and  other 
patient  factors  as  a  means  to  compare  quality  of 
care  among  hospitals  each  year  (Smith,  Pine,  Bailey, 
and  others,  1991).  In  the  late  1980s,  the  JCAH 
began  developing  a  set  of  clinical  indicators  to  be 
used  for  comparing  quality  of  care  among  hospitals 
(Joint  Commission  on  Accreditation  of  Healthcare 
Organizations,  1990).  Widespread  testing  of  these 
measures  has  now  begun.  Once  these  measures  are 
evaluated,  the  JCAH  will  require  hospitals  to  report 
their  performance  on  these  indicators  and  to  receive 
feedback  comparing  their  performance  with  that  of 
other  similar  hospitals.  The  comparisons  are 
expected  to  stimulate  hospitals  to  improve  in  order 
to  match  the  performance  of  their  peers. 
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Advances  in  computer  technology.  Quality  mea- 
surement by  external  agencies  in  the  1990s  will  be 
conducted  on  a  large  scale  by  use  of  data  bases  and 
computerized  methods  of  applying  quality  of  care 
criteria.  The  original  vision  of  quality  measurement 
is  getting  easier  and  cheaper  to  realize  as  powerful 
computers  become  portable  and  as  more  and  more 
clinical  data  are  available  on  computer.  The  PRO 
program  has  developed  a  Uniform  Clinical  Data  Set 
(Wilensky,  1991)  to  permit  more  sophisticated  algo- 
rithmic analyses  of  process  data  and  more  sophisti- 
cated case  mix  adjustments  for  outcomes  data.  The 
Department  of  Health  and  Human  Services  has 
announced  its  support  for  a  long-term 
project  leading  to  development  of  a  computerized 
patient  medical  record  (Institute  of  Medicine,  1991). 
Its  use  could  both  directly  improve  the  delivery  of 
care  and  make  it  easier  to  measure  quality  and 
manage  QI. 

Effectiveness  of  quality  assurance.  We  can  sum 

up  the  experience  with  QA  in  the  1970s  and  1980s 
by  saying  that  much  was  learned  about  quality  mea- 
surement, but  dramatic  gains  in  quality  of  care  did 
not  result.  Both  internal  and  external  QA  was 
resisted  by  providers.  Physicians,  especially,  clung  to 
the  professional  model,  which  holds  them  account- 
able only  to  their  own  patients  and  to  their  immedi- 
ate colleagues.  A  distaste  for  QA  seemed  justified 
because,  when  implemented  within  bureaucratic 
health  care  institutions,  it  had  little  positive  effect. 
This  shortfall  should  not  be  surprising  because  QA 
effected  little  change  in  industries  with  bureaucratic 
structures.  However,  other  industries,  learning  from 
the  example  of  Japanese  high-technology  industries, 
began  restructuring  themselves  in  ways  that  permit- 
ted astonishing  gains  in  product  quality.  We  will 
turn  now  to  examine  the  movement  in  industry  for 
CQI/TQM  and  how  it  has  begun  to  energize  the 
quality  movement  in  health  care. 

Industrial  Perspective  on  Quality 

To  understand  the  industrial  perspective  on 
quality,  we  need  to  think  back  to  the  switch  in 
manufacturing  technique  from  hand-made  to 
machine-made.  Hand-made  goods  are  produced  by 
a  craftsman  who  makes  all  the  parts  and  fits  each 
part  to  the  others  individually.  Machine-made  goods 
are  produced  by  many  different  workers,  each 
making  just  one  part,  while  other  workers  assemble 
the  parts  into  a  manufactured  product.  For 


machine-made  products,  parts  of  the  same  type 
must  be  interchangeable,  and  parts  of  different 
types  must  conform  to  precise  specifications  so 
components  will  fit  one  another  without  hand 
finishing.  The  production  process  cannot  permit 
variation — it  must  produce  standardized  parts  or  it 
will  fail.  Thus  manufacturers'  definition  of  quality 
came  to  be  "conformance  to  specifications." 

A  factory  has  many  workers  on  its  assembly  line; 
their  activities  are  interdependent  and  have  to  be 
coordinated  if  the  work  is  to  be  productive.  There- 
fore, industrialization  introduced  systems  for  pro- 
duction. A  factory  has  many  systems,  one  for  each 
"production  process."  A  system  includes  inputs, 
workers,  and  outputs:  each  worker  receives  inputs, 
processes  them,  and  passes  them  on  as  outputs. 

To  design  a  system,  managers  must  choose  the 
desired  outputs,  secure  a  supply  of  appropriate 
inputs,  and  design  a  production  process  known  to 
produce  these  outputs  from  the  given  inputs.  Each 
worker  must  then  be  trained  to  operate  according  to 
the  system  design.  One  worker  "hands  off"  work  to 
the  next  in  the  production  line.  The  system  cannot 
accommodate  workers  interjecting  their  own  ideas  of 
how  to  do  the  job  because  that  will  ruin  the  system; 
workers  must  do  just  what  they  are  told  to  do. 

System  variation  and  system  error.  Complex 
systems  are  prone  to  variation  from  one  production 
run  to  another.  The  more  complex  the  system,  the 
greater  the  chance  of  variation.  Since  quality 
depends  on  conformance  to  specifications  or  the 
absence  of  variation,  any  variation  from  product 
specifications  is  an  error.  So  "system  error"  is  the 
amount  of  chance  variation  inherent  in  a  system,  and 
it  is  an  attribute  of  a  particular  system.  This  kind  of 
variation  is  known  by  several  other  terms:  random 
variation,  common-cause  variation,  or  systemic 
variation  (Berwick,  1991;  Kritchevsky  and  Simmons, 
1991;  Laffel  and  Blumenthal,  1989;  Plsek,  1990). 

By  measuring  the  performance  of  a  system 
repeatedly,  we  can  detect  the  level  of  error  inherent 
in  the  system  as  it  is  designed.  While  error  rates  will 
vary  over  the  short  term,  over  the  long  term  they 
remain  within  statistically  predictable  limits  that  are 
characteristic  for  a  given  system.  Poorly  designed 
systems  will  have  high  error  rates. 

Reducing  system  error  by  improving  system 
design  has  a  large  effect  on  quality  because  it  affects 
all  future  runs  using  this  system.  However,  we  can- 
not ever  eliminate  all  system  error  because  of  limits 
on  the  resources  that  can  be  spent  on  perfecting 
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system  design  and  operation.  Workers  cannot  be 
blamed  for  errors  inherent  in  the  system;  these  can 
affect  any  worker  no  matter  how  competent, 
because  they  are  not  under  the  worker's  control. 

Failure  to  adhere  to  the  system  also  can  result  in 
unwanted  variation.  Events  that  are  not  inherent 
parts  of  the  intended  system  may  cause  variations 
(errors),  which  are  called  special-cause  or  extra- 
systemic  variations. 

If  we  measure  the  performance  of  a  system 
repeatedly,  we  can  infer  that  an  extrasystemic  cause 
is  at  work  if  the  observed  variation  goes  beyond 
the  bounds  of  the  usual,  short-term,  common 
cause/random  variation.  Extrasystemic  variation 
tends  to  cluster  in  association  with  whatever  is  the 
special  cause  (Kritchevsky  and  Simmons,  1991). 
A  special  cause  may  be  related  to  a  person,  a  place, 
or  a  time.  The  variation  will  recur  whenever  the 
extrasystemic  cause  is  at  work.  If  the  extrasystemic 
cause  is  a  persistent  influence,  "out-of-bounds" 
variation  will  persist,  or  if  the  systemic  cause  is  a 
steadily  growing  influence,  the  variation  will  get 
steadily  worse.  To  detect  a  special  cause,  we  exam- 
ine the  pattern  of  occurrence  of  the  variation. 

Extrasystemic  variation  occurs  when  one  has  lost 
control  of  the  system  (Kritchevsky  and  Simmons, 
1991).  To  restore  control,  one  needs  to  remove  the 
cause  of  extrasystemic  variation;  for  example,  if  it  is 
an  incompetent  worker,  he  or  she  must  either  be 
retrained  or,  if  recalcitrant,  replaced.  If  a  defective 
chemical  is  present  in  a  clinical  laboratory  autoana- 
lyzer,  it  must  be  replaced.  Getting  rid  of  special- 
cause  variation  can  be  thought  of  as  maintaining  the 
system  through  activities  known  as  "quality  control." 

If  the  system  frequently  goes  out  of  control 
because  of  workers'  failures  to  adhere  to  the  system, 
it  is  not  robust.  Managers  then  redesign  the  system 
to  tolerate  a  greater  range  of  worker  competence 
and  thus  facilitate  maintaining  control.  For  instance, 
airline  engineers  redesign  cockpit  instruments  to 
make  pilot  error  less  likely.  This  is  called  human 
engineering.  Note  that  a  system  cannot  be  rede- 
signed by  workers  within  it — redesign  must  come 
from  those  responsible  for  system  management. 

The  industrial  model  applied  to  health  care: 
Definitions  of  quality.  In  industry,  as  in  health  care, 
quality  is  defined  differently  for  different  purposes  and 
from  different  perspectives.  Garvin  (1984)  grouped  the 
definitions  and  described  them  as  follows. 

In  the  concept  of  "product-based"  quality, 
standards  define  categories  of  quality,  with  more  of 
some  ingredient  associated  with  higher  quality 


(structure).  For  instance,  a  type  of  rug  is  graded  as 
to  quality  by  the  number  of  knots  per  square  inch. 
More  raw  materials  used  imply  better  quality,  and 
better  quality  costs  more.  Familiar  examples  are  the 
designations  good,  better,  and  "Sears  best"  in  the 
well-known  catalog.  Customers  choose  the  products 
they  can  afford. 

For  a  given  product,  manufacturers  define  qual- 
ity as  "conformance  to  specifications"  or  "doing  it 
right  the  first  time"  (process).  More  quality  in  man- 
ufacturing means  lower  cost  because  reworking  the 
defective  product  and  fixing  mistakes  are  avoided. 
This  is  the  logic  behind  the  claim  that  poor  quality 
costs  money. 

The  TQM  approach  favors  "user-based"  defini- 
tions of  quality:  these  state  that  quality  consists  of 
meeting  the  needs  and  expectations  of  customers.  In 
other  words,  the  definitions  are  outcome  oriented. 

Value-based  definitions  begin  with  an  assump- 
tion of  limited  resources;  they  define  quality  as  the 
best  performance  that  one  can  afford  to  buy. 

What  is  the  relevance  of  these  ideas  to  health 
care?  Different  health  care  products  that  specify 
levels  of  comfort  and  convenience,  what  Donabe- 
dian  calls  the  amenities  of  care  (Donabedian,  1988), 
have  long  been  offered  at  different  prices.  For 
instance,  a  health  insurance  policy  covers  only  a 
shared  hospital  room;  a  patient  who  desires  a 
private  room  must  pay  more.  When  we  consider 
technical  quality  (the  coordination  of  knowledge, 
skill,  and  resources  to  improve  patients'  health),  it  is 
different.  Most  Americans  reject  the  idea  that  more 
or  less  technical  quality  of  care  should  be  offered  at 
different  prices.  It  strikes  at  the  egalitarian  ideal 
that  all  lives  are  of  equal  worth  and  therefore  equally 
worth  saving.  Instead,  technical  quality  is  viewed  as 
a  kind  of  public  utility  to  be  offered  at  one  level  of 
quality  (the  best!)  and  at  one  price.  JCAHO  stan- 
dards for  hospital  facilities  come  closest  to  the  con- 
cept of  product-based  quality  in  that  they  specify 
the  equipment,  facilities,  and  staffing  of  a  hospital. 
In  keeping  with  the  "public  utility"  notion,  they 
specify  one  level  of  product:  an  accredited  facility. 

It  is  easy  to  see  that  the  manufacturer's  definition 
of  quality  fits  well  in  the  delivery  of  health  services. 
In  producing  a  computed  tomography  (CT)  scan 
that  is  interpreted  and  acted  upon,  a  wound  dressing 
change,  a  dietary  department  bed-day,  or  a  course  of 
intravenous  therapy,  good  quality  requires  doing  it 
right  the  first  time.  It  is  also  true,  as  in  industry,  that 
doing  it  right  the  first  time  will  cost  less. 
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Can  we  push  the  definition  into  physicians' 
work?  A  physician  practice  guideline  is  a  kind  of 
specification  of  the  right  things  to  do.  Once  we  have 
specified  a  practice  guideline,  we  can  design  a  pro- 
duction system  to  make  it  easy  to  complete  the  care 
indicated  by  the  guideline.  The  production  system 
will  not  force  conformance  with  the  guideline  when 
it  does  not  apply;  it  will  just  make  it  easier  to  follow 
the  guideline.  This  production  system  will  consist  of 
the  fewest  and  simplest  possible  sequences  of  activi- 
ties that  support  consistent  performance  of  the 
guideline.  In  order  for  the  system  to  operate  well, 
all  staff  of  the  health  care  facility  or  hospital  should 
be  trained  for  their  part  in  the  production  system. 

Designing  systems  to  support  well-standardized 
physician  activities  is  clearly  desirable  in  health 
care.  For  example,  certain  activities  of  well-child 
care  are  recommended  by  the  American  Academy 
of  Pediatrics.  In  a  study  of  eight  pediatric  practices 
(Palmer,  Strain,  Maurer,  and  Thompson,  1984),  we 
observed  that  those  with  well-specified  routines  for 
accomplishing  these  activities  did  better  at  complet- 
ing the  prescribed  course  of  well-child  care.  Physi- 
cians at  the  Harvard  Community  Health  Plan  have 
begun  a  comprehensive  approach  to  this  kind  of 
system  engineering.  Physicians  in  the  health  care 
plan  establish  algorithmic  guidelines  for  managing 
common  patient  conditions;  then  management 
provides  the  system  support  to  enable  physicians  to 
comply  with  their  guideline  with  the  least  possible 
waste,  rework,  and  complexity  (Gottlieb,  Margolis, 
and  Schoenbaum,  1990).  Can  we  imagine  extending 
this  thinking  to  widespread  implementation  of  the 
guidelines  emerging  from  the  AHCPR-supported 
panels?  Why  not? 

User-based  definitions  of  quality  are  now  used 
in  health  care.  Many  HMOs  and  hospitals  routinely 
measure  patients'  satisfaction  with  services  given 
and  seek  input  on  the  needs  and  expectations  of 
enrollees  and  potential  enrollees  (Nelson,  Rubin, 
Hays,  and  Meterko,  1990;  Nelson,  Ware,  and 
Batalden,  1990).  A  user-based  definition  of  quality 
is  implied  by  programs  to  elicit  patients'  preferences 
in  making  decisions  about  care  (Greenfield,  1988). 
Another  example  is  the  growing  practice  of  collect- 
ing data  on  patients'  outcomes  after  care  and  using 
this  information  to  redirect  the  processes  of  care 
(Nelson,  Landgraf,  Hayes,  and  others,  1990).  An 
especially  promising  aid  to  meeting  patients'  needs 
is  the  routine  use  of  surveys  of  patients'  functional 
health  status.  These  are  now  available  in  short  and 


easily  used  formats  (Nelson,  Ware,  and  Batalden, 
1990;  Stewart,  Hayes,  and  Ware,  1988). 

Last,  value-based  definitions  of  quality  have  an 
obvious  place  in  health  care.  In  the  1980s,  public 
policy  was  to  fix  prices  and  control  volume  of 
services  through  external  review.  For  instance,  the 
Medicare  program  sets  a  single  fixed  price  for  a  unit 
of  hospital  care  or  physician  service.  The  value- 
based  definition  of  quality  now  applies:  we  seek  the 
highest  quality  of  this  unit  of  care  given  the  fixed 
price.  This  is  how  we  obtain  value  for  money. 

Quality  management  activities  in  health  care. 
The  new  version  of  quality  management  brings  with 
it  a  new  "corporate  culture"  wherein  all  employees, 
professional  and  nonprofessional,  strive  for  quality 
in  their  work.  Top  management  must  exert  leader- 
ship to  this  end.  New  structures  are  required:  QI 
activities  are  coordinated  by  a  steering  committee 
but  are  conducted  by  QI  teams  within  units  of  the 
health  care  organization.  The  teams  spin  off  work 
groups  to  study  and  solve  specific  problems.  For 
most  health  care  institutions  this  will  require  a 
breakdown  of  existing  hierarchies  into  a  more 
flexible  and  interdisciplinary  style  of  organization. 

Since  quality  is  defined  as  meeting  the  needs 
and  expectations  of  customers,  a  dialog  must  begin 
between  customers  and  suppliers.  Patients  are  obvi- 
ously customers,  but  colleagues  and  coworkers  to 
whom  one  supplies  services  also  are  seen  as  "inter- 
nal customers."  Managers  must  discover  what  both 
kinds  of  customers  define  as  the  key  quality  charac- 
teristics for  performance  of  specific  services.  For 
example,  within  hospitals,  managers  can  discover 
what  gastroenterologists  need  and  expect  from 
radiologists  and  what  nursing  needs  and  expects 
from  pharmacy.  Also,  through  surveys  or  focus 
groups,  health  care  plans  can  discover  what  their 
enrollees  need  and  expect.  Measuring  quality  char- 
acteristics and  improving  performance  through 
system  design,  maintenance,  and  improvement 
becomes  the  work  of  the  QI  teams. 

Introduction  of  a  new  technology  or  a  new 
practice  guideline  should  trigger  a  phase  of  quality 
design.  For  instance,  when  a  new  defibrillator  is 
introduced,  or  surgeons  start  doing  gallbladder 
surgery  by  laparoscopy,  a  system  for  safe  and  prompt 
implementation  should  be  specified,  and  all  staff 
concerned  should  be  trained  to  fulfill  their  roles. 

Performance  of  systems  already  in  place  must 
be  studied  to  see  how  they  are  constructed  and 
whether  they  are  "in  control,"  that  is,  whether  there 
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is  any  inconsistency  in  the  way  the  work  is  done 
(Berwick,  1991;  Kritchevsky  and  Simmons,  1991; 
Laffel  and  Blumenthal,  1989).  How  often  is  valuable 
time  in  the  cardiac  catheterization  laboratory 
wasted  because  patients  are  not  transported  from 
their  inpatient  unit  on  time?  How  does  on-time 
performance  vary?  If  a  system  is  out  of  control — 
that  is,  performance  is  unstable  and  unpredictable — 
it  must  be  brought  under  control.  If  a  system  is  in 
control,  its  performance  is  measured,  and  if  the 
level  of  performance  is  unacceptable,  the  system 
must  be  redesigned. 

Redesign  of  systems  is  expensive  and  disruptive 
in  the  short  term.  Therefore,  before  a  QI  project  is 
launched,  managers  must  be  sure  that  they  have 
correctly  diagnosed  the  problem.  This  is  why  QI 
emphasizes  the  study  of  causes  of  system  errors.  By 
probing  further  back  into  a  chain  of  causation,  we 
may  locate  a  simple  but  powerful  solution.  It  is  par- 
ticularly important  to  distinguish  between  systemic 
and  extrasystemic  causes  because  trying  to  fix  a 
system  on  the  basis  of  a  wrong  diagnosis  can  make 
performance  worse. 

In  industry,  misguided  efforts  to  fix  systems  are 
called  "tampering"  (Plsek,  1990).  It  is  tampering  to 
discipline  a  worker  for  an  error  caused  by  the 
system — that  is,  to  treat  a  systemic  cause  as  extra- 
systemic.  It  is  also  tampering  to  change  a  whole 
system  for  care  when  one  person's  inability  to  do  his 
or  her  job  is  the  cause  of  poor  performance — that 
is,  treating  an  extrasystemic  cause  as  if  it  is  systemic. 
It  is  also  tampering  to  react  to  random  variation  in 
system  performance  by  changing  the  system.  Most 
health  care  workers  encounter  frequent  examples 
of  each  of  these  forms  of  tampering  in  health  care 
organizations  (Berwick,  1991).  Tampering  wastes 
effort  and  can  add  complexity  to  systems  if  extra 
policies  and  procedures  are  piled  on  in  an  effort  to 
patch  up  inadequate  systems.  Increased  complexity 
adds  to  costs  and  increases  the  risk  of  error. 

Collecting  and  interpreting  quality  data:  Quality 
management  tools.  Industry  uses  a  wealth  of  simple 
but  elegant  tools  for  displaying  and  analyzing 
quality  measurements  and  for  helping  workers  draw 
upon  their  own  rich  experiences  to  analyze  causes 
of  problems  in  their  work  environment  (Plsek, 
1990).  Leaders  in  many  health  care  organizations 
are  already  adopting  them.  For  instance,  flow  charts 
of  a  sequence  of  activities,  checksheets,  or  brain- 
storming sessions  can  be  used  to  identify  problems 
affecting  performance  of  a  system.  Pareto  charts  or 


pie  charts,  cause-and-effect  diagrams  (also  called 
fishbone  diagrams),  run  charts,  and  the  "is/is  not 
matrix"  are  tools  used  both  to  identify  problems 
and  to  begin  probing  their  causes.  Familiar  tech- 
niques such  as  the  histogram,  scattergram,  control 
chart,  bar  chart,  stem  and  leaf  plots,  and  graphs 
are  used  for  further  pinpointing  of  causes.  Many  of 
these  tools  are  specifically  designed  to  help  individ- 
uals with  little  formal  analytic  training  to  under- 
stand and  interpret  data. 

Notice,  however,  that  these  are  not  themselves 
tools  for  measurement.  Each  industry  must  produce 
methods  of  measurement  specific  to  its  own  work.  In 
order  to  adopt  TQM,  health  care  organizations  will 
need  to  build  on  the  techniques  for  quality  measure- 
ment developed  during  the  era  of  QA  in  health  care. 

Quality  management  in  health  care:  Will  it  be 
used?  Since  health  care  is  now  often  thought  of 
as  an  industry,  it  is  not  surprising  that  TQM  as 
practiced  among  high-technology  industries  is  being 
applied  to  health  care.  This  movement  has  been 
encouraged  by  big  corporations:  those  who  adopt 
quality  management  in  their  own  businesses  are 
increasingly  interested  in  controlling  health  care 
costs  for  their  organizations.  The  Federal  Govern- 
ment also  has  adopted  policies  that  promote  an 
industrial  approach  to  health  care. 

Will  the  transfer  of  quality  management  ideas 
from  industry  to  health  care  work  at  this  time? 
Health  care  differs  from  other  industries  in  ways 
that  may  hinder  the  adoption  of  TQM  (Garvin, 
1990).  As  I  have  noted,  the  connection  between 
inputs  and  outputs  in  health  care  is  not  well  under- 
stood; most  customers  are  unable  to  judge  quality, 
and  measures  of  performance  of  most  production 
systems  are  still  lacking. 

On  the  other  hand,  managing  health  care  fits  an 
industrial  rather  than  a  bureaucratic  model  in  some 
important  respects.  Industry's  emphasis  on  meeting 
customers'  needs  fits  the  professional  model's 
emphasis  on  accountability  to  patients  better  than 
the  bureaucratic  view  of  patients  as  passive  bene- 
ficiaries. The  flexibility  of  industrial  models  of 
management,  at  least  in  technologically  complex 
industries,  is  probably  better  suited  to  health  care 
than  the  bureaucratic  model  of  separate  hierarchies 
of  authority.  Even  physician-manager  relationships 
may  be  easier  in  the  industrial  model  if  physicians 
see  themselves  as  business  partners  of  management. 
Physicians  in  private  practice  have  always  acted  as  a 
modified  form  of  entrepreneur.  The  partner  role  is 
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likely  to  be  more  attractive  than  the  ambiguous 
position  that  physicians  occupy  in  a  bureaucratic 
model,  as  subservient  to  administrators  on  adminis- 
trative matters  but  autonomous  in  clinical  judgment. 
However,  autonomy  in  clinical  judgment  is  an  area 
of  potential  conflict  in  the  industrial  model  because 
the  industrial  model  seeks  standardization  of  work 
whenever  possible,  and  physicians  (advisedly,  to 
some  extent)  tend  to  resist  it. 

As  we  noted  earlier,  the  innovators  who  intro- 
duced QA  into  health  care  during  the  1970s  molded 
it  upon  the  then-prevailing  industrial  model.  In 
implementation,  QA  was  transplanted  into  a 
bureaucratic  environment,  which  doomed  it  to 
failure.  The  chance  that  quality  management  will 
succeed  in  health  care  may  be  better  this  time 
around  for  several  reasons.  First,  the  switch  from 
the  bureaucratic  to  the  quasi-competitive  environ- 
ment provides  positive  motivation  for  quality 
management;  it  is  now  in  the  provider's  self-interest 
and  must  be  done  for  survival,  not  merely  to  please 
an  outside  regulator.  Second,  methodology  for 
quality  measurement  has  advanced.  Detailed,  rapid, 
real-time  assessment  tools  suitable  for  use  in  quality 
management  in  health  care  are  becoming  available. 
Third,  experts  from  quality  management  in  industry 
are  bringing  their  skills  and  experience  to  the  health 
care  field.  Fourth,  physicians  are  motivated  to 
improve,  according  to  their  internalized  professional 
imperative,  if  they  are  convinced  that  patients' 
welfare  is  truly  at  stake.  Credible  measurement  of 
quality  can  persuade  physicians  that  improvement 
is  required.  Fifth,  TQM  is  likely  to  make  great 
improvements  in  the  health  care  systems  on  which 
physicians  depend  to  do  their  work.  This  factor  may 
persuade  physicians  to  "buy  in." 

Few  doubt  that  quality  management  will  apply 
to  services  (material  flow,  patient  flow,  information 
flow  in  health  care).  Less  clear  is  the  potential  use 
of  quality  management  for  standardizing  clinical 
work,  then  managing  it  to  produce  quality.  The 
clinical  guidelines  initiative  at  the  Harvard  Commu- 
nity Health  Plan  may  point  the  way  (Gottlieb, 
Margolis,  and  Schoenbaum,  1990). 

Role  of  External  Agencies  in  Quality 
Improvement 

How  will  external  review  change  in  the  era  of 
the  industrial  model?  I  will  briefly  examine  how 
external  review  strategies  have  evolved  to  date.  In 


the  professional  model,  State  governments  protect 
the  public  from  incompetent  or  unscrupulous 
providers  through  licensing  entrants  to  the  profes- 
sion and  disciplining  or  withdrawing  licenses  from 
providers  who  violate  professional  standards.  This 
is  an  example  of  a  purely  regulatory  approach  to 
improving  quality  for  the  public;  it  fits  the  bureau- 
cratic model  of  enforcing  impersonal  rules.  This 
activity  will  always  be  needed. 

A  regulatory  approach  also  was  made  mandatory 
for  the  PROs  in  their  second  "scope  of  work" 
(Lohr,  1990).  PROs  reviewed  hospital  records  using 
generic  screening  criteria  to  detect  adverse  events. 
For  each  adverse  event  thus  detected,  a  physician 
reviewer  was  required  to  determine  whether  the 
adverse  event  resulted  from  a  "gross  and  flagrant," 
or  a  "substantial"  violation  of  professional  stan- 
dards; these  terms  were  strictly  defined  in  the 
"scope  of  work."  For  a  single  gross  and  flagrant  vio- 
lation 

or  for  five  or  more  substantial  violations,  the  PRO 
was  required  to  initiate  proceedings  to  sanction  the 
physician.  In  other  words,  no  action  other  than 
punishment  was  permitted. 

With  time,  as  accused  physicians  hired  lawyers 
to  defend  their  rights,  a  lengthy  "due  process" 
evolved  for  PRO  sanction  proceedings.  During  this 
process,  it  often  emerged  that  a  physician  against 
whom  sanctions  were  initiated  had  failed  to  keep  up 
with  medical  advances  and  was  following  practices 
learned  years  ago  that  were  no  longer  appropriate. 
PRO  activities  were  amended,  at  the  urging  of  the 
PROs  themselves,  to  permit  requiring  a  physician 
with  "violations"  to  undertake  a  "QI  plan"  such  as 
undergoing  remedial  training  or  obtaining  consul- 
tation in  future  cases.  In  other  words,  the  strictly 
bureaucratic  approach  was  leavened  with  elements 
of  a  professional  approach  to  an  erring  colleague. 
Nevertheless,  PROs  are  perceived  by  most  physi- 
cians as  taking  an  entirely  negative  approach  to 
quality:  that  of  finding  "bad  apples"  and  removing 
them  from  the  program. 

A  regulatory  approach  is  demanded  by  some 
consumer  advocacy  groups.  Their  campaigns 
imply  that  improvement  in  quality  is  measured  by 
numbers  of  license  removals  and  of  physicians 
sanctioned  (Lohr,  1990). 

When  government  first  began  to  pay  for  health 
care,  it  also  adopted  a  regulatory  approach  to  the 
issue  of  unnecessary  care.  Physicians  were  required 
to  certify  that  services  provided  were  needed. 
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Adjudicating  the  need  for  services  was  viewed  like 
any  other  situation  in  which  bureaucracies  adminis- 
ter benefits;  it  was  a  matter  of  applying  impersonal 
decision  rules.  Violation  of  the  rules  resulted  in 
denial  of  reimbursement  for  the  service.  Private 
third-party  payers  behaved  similarly.  As  use  and 
costs  of  health  services  grew  dramatically,  both 
public  and  private  third-party  payers  redoubled 
their  regulatory  efforts,  so  that  a  bewildering  num- 
ber of  external  agencies  now  review  providers'  work 
in  order  to  deny  payment  for  unnecessary  care. 

Proponents  of  quality  management  attack  the 
regulatory  approach  as  counterproductive.  Regula- 
tion violates  one  of  Deming's  principles  of  quality 
management:  "drive  out  fear"  (Walton,  1986).  The 
regulatory  approach  invites  providers  to  hide  from 
or  subvert  quality  measurement.  At  best,  it  only 
cuts  the  low  end  off  the  distribution  curve  of  quality, 
whereas  quality  management  can  move  the  average 
level  of  quality  to  a  higher  level,  a  greater  gain  in 
quality  (Berwick,  1989).  Furthermore,  complying 
with  the  multiplicity  of  external  review  require- 
ments is  costly.  Advocates  of  quality  management 
assert  that  the  end  result  of  a  purely  regulatory 
approach  is  inevitably  more  expensive,  but  care  is 
unimproved.  Many  health  care  professionals  are 
attracted  to  this  argument. 

Externa]  agencies  and  quality  management. 
Advocates  of  quality  management  argue  that  QI 
should  be  an  internal  affair  that  is  self-motivated, 
driven  by  the  desire  to  attract  customers.  They 
contrast  this  with  the  bureaucratic  approach  of 
"do  it  because  we  say  so."  Nevertheless,  we  must 
recognize  that  the  power  of  quality  management  in 
industry  derives  from  the  choices  of  customers  who 
can  distinguish  degrees  of  quality  in  a  product.  The 
desire  to  please  customers  and  increase  market 
share  will  not  drive  QI  unless  customers  buy  the 
better  product.  High-technology  companies  in  the 
United  States  did  not  adopt  quality  management 
until  Japanese  companies  with  higher-quality  prod- 
ucts entered  the  market  and  won  over  customers 
(White,  Patterson,  and  Ingrassia,  1992).  While  some 
large  hospitals  and  HMOs  are  adopting  quality 
management  wholeheartedly,  it  is  doubtful  that 
this  drive  for  excellence  will  spread  to  all  providers, 
or  even  persist  in  these  initially  enthusiastic  organi- 
zations, if  customers  do  not  reward  the  better 
providers  by  giving  them  more  business. 

It  is  not  easy  for  customers  to  choose  health  care 
on  the  basis  of  quality.  Many  factors  enter  into  a 


decision  about  where  to  get  health  care.  Some 
patients  may  value  ease  of  access;  to  others  a  sense 
of  trust  and  interpersonal  ease  are  more  important. 
Patients  with  a  long-established  relationship  with  a 
physician  or  hospital  often  resist  a  change,  even  if 
the  provider  is  actually  dangerous  in  the  technical 
sense.  A  major  obstacle  is  that  limited  aspects  of 
quality  are  directly  visible  to  patients;  others, 
especially  technical  excellence,  are  not. 

Federal  policy  and  some  consumer  advocates 
have  pressed  for  better  measurement  of  quality 
and  dissemination  of  data  on  quality  of  providers 
to  assist  consumers'  choice  (Office  of  Technology 
Assessment,  1988).  Publication  of  such  information 
is  not  likely  to  prove  practical  at  the  level  of  the 
individual  physician  because  it  is  so  difficult,  both 
financially  and  logistically,  to  measure  quality  at 
that  level.  Measurement  of  quality  at  the  level  of 
the  individual  is  also  confounded  because  many 
physicians  may  be  involved  in  any  one  patient's 
care,  and  in  examining  outcomes,  it  is  impossible 
to  disentangle  the  effects  of  the  multiple  providers. 
It  is  much  easier  to  measure  the  quality  of  a 
group  or  of  a  provider  organization  than  of  an 
individual  physician. 

Another  limitation  to  consumer  choice  of 
health  care  according  to  quality  is  the  fact  that 
most  consumers  have  difficulty  understanding 
quality  measures.  Even  if  they  could  understand, 
they  are  not  likely  to  devote  the  time  and  effort 
required  to  make  an  informed  choice  when  they 
are  healthy,  and  a  patient  with  urgent  need  for 
care — for  example,  one  in  imminent  need  of 
neurosurgery — is  likely  to  be  too  ill  and  upset  to 
go  "comparison  shopping"  for  a  physician  with  the 
needed,  highly  specialized  skills. 

For  these  reasons,  it  has  become  common  for 
health  care  shopping  to  be  done  at  a  more  aggre- 
gate level  than  the  individual  provider  and  patient. 
Increasingly,  care  is  given  by  managed-care  plans 
such  as  HMOs,  preferred  provider  organizations 
(PPOs),  and  independent  practice  associations 
(IPAs).  Third  parties  who  subsidize  or  pay  for 
health  insurance,  including  government  and  large 
corporations,  limit  the  choice  of  health  care  plans 
for  their  beneficiaries  and/or  employees.  These 
bulk  purchasers  contract  selectively  with  health 
care  plans  to  get  care  at  a  lower  cost. 

Such  an  arrangement  is  not  a  bargain  if  patients 
do  not  get  value  for  their  money.  Therefore,  bulk 
purchasers  would  prefer  to  select  health  care  plans 
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on  the  basis  of  high  quality.  Once  a  purchaser  has 
contracted  for  service,  and  particularly  if  a  hard 
bargain  was  driven  on  price,  the  health  care  plan 
may  be  tempted  to  enhance  its  financial  status  by 
skimping  on  services  and  discouraging  sicker  patients 
from  enrolling.  Thus  selective  contracting  ("buying 
right")  must  include  ongoing  and  accurate  measure- 
ment of  quality.  The  purchaser  needs  a  limited  num- 
ber of  easy-to-understand  measurements  of  quality 
that  sum  up  the  health  care  plan's  performance. 

Customer-supplier  dialog.  This  scenario  predicts 
that  in  the  future  employers  and  governments  will 
contract  for  health  services  on  the  basis  of  value  for 
money.  In  the  purely  competitive  model,  a  health 
care  plan  might  not  be  selected  by  the  purchaser  in 
a  given  year  because  other  health  care  plans  provide 
better  value. 

Industry  recognizes  that  frequent  changes  of 
supplier  are  disruptive  and  ultimately  counterpro- 
ductive; this  would  apply  particularly  in  health 
care,  where  patients  may  have  long-term  trusting 
relationships  with  their  doctor  or  hospital.  Japanese 
industry  has  adopted  a  solution  to  this  problem 
called  "keiretsu,"  which  can  be  translated  as  coop- 
erative links  or  consortia  (Kelly,  Port,  Tree,  and  oth- 
ers, 1992).  Keiretsu  come  in  vertical  and  horizontal 
forms.  A  vertical  keiretsu  is  a  form  of  "customer- 
supplier"  dialog.  In  contrast  to  the  tough  and  adver- 
sarial approach  to  procuring  services  usually 
adopted  in  American  industries,  a  keiretsu  is  a  long- 
term,  cooperative  partnership  between  a  supplier 
and  purchaser  in  which  the  two  collaborate  to 
improve  the  quality  of  materials  supplied.  In  this 
arrangement,  measurements  of  quality  are  still 
needed,  but  they  are  used  in  negotiations  between 
the  customer  and  supplier. 

What  form  will  quality  measurements  used  in 
customer-supplier  negotiations  for  health  care  take? 
Obviously  it  is  too  early  to  know,  but  the  new  direc- 
tions are  becoming  clearer.  In  January  1992,  HCFA 
announced  its  intention  to  begin  working  collabora- 
tively with  provider  groups  through  the  PROs  to 
improve  quality  in  the  Medicare  program  (Jencks 
and  Wilensky,  1992).  HCFA  will  be  developing  mea- 
surements of  quality  using  data  from  its  National 
Claims  History  File  and  by  abstracting  data  from 
hospital  records  into  the  Uniform  Clinical  Data  Set. 
HCFA  intends  to  construct  quality  measurements 
differently  than  in  the  past.  Previously,  HCFA  strat- 
egy was  to  detect  poor  performance  outliers;  in  its 
new  HCQII,  HCFA  will  use  the  industrial  technique 


of  benchmarking  by  identifying  the  range  of  perfor- 
mance, focusing  on  the  best  performance,  and 
encouraging  providers  to  match  it. 

The  new  quality  indicators  being  developed  by 
JCAHO  also  might  be  used  in  customer-supplier 
negotiations  concerning  quality  (Joint  Commission 
on  Accreditation  of  Healthcare  Organizations, 
1991).  They  will  provide  valid  comparisons  among 
hospitals  on  a  range  of  indicators  covering  all  major 
clinical  departments. 

There  is  a  different  type  of  assessment  of  quality, 
not  truly  a  measurement,  that  could  be  used  in  cus- 
tomer-supplier negotiation.  It  is  the  assessment  of 
internal  efforts  for  QI.  JCAHO  is  gradually  revising 
its  accreditation  requirements  to  encourage  CQI 
(Joint  Commission  on  Accreditation  of  Healthcare 
Organizations,  1991).  There  are  two  industrial  mod- 
els for  assessing  internal  QI  efforts:  one  is  the 
esteemed  Malcolm  Baldrige  Award  in  the  United 
States  (Berwick,  1990);  the  other  is  the  ISO  9000 
certificate  required  for  companies  trading  in  the 
European  Common  Market  (Hockman,  1991).  In 
the  future,  JCAHO  accreditation  standards  may 
come  to  resemble  the  criteria  for  these  awards. 

Research  for  Quality  Improvement 

Research  on  QA  in  health  care  began  in  the 
professional  era.  Some  date  its  origin  to  Florence 
Nightingale's  analysis  of  patterns  of  infection 
among  soldiers  injured  in  the  Crimean  War 
(Lohr  and  Brook,  1984).  Most  scholars  trace  QA 
research  in  the  United  States  to  Ernest  Codman's 
studies  of  the  "end  results"  of  surgery  in  1911 
(Reverby,  1981).  Led  by  such  pioneers  as  Paul 
Lembcke,  the  pace  of  research  quickened  in  the 
1950s  and  1960s  (Silver,  1990).  Research  in  QA  was 
often  funded  by  physician  organizations  seeking  to 
maintain  quality  by  expanding  existing  methods  for 
voluntary  collegial  activities. 

In  the  1970s  and  1980s,  with  QA  mandatory 
within  Medicare,  research  began  to  emphasize  mea- 
surement on  a  larger  scale  and  use  it  for  QI.  Much 
innovative  work,  including  the  Experimental  Med- 
ical Care  Review  Organization  program,  was  funded 
by  the  National  Center  for  Health  Services  Research 
and  Health  Care  Technology  Assessment,  AHCPR's 
predecessor  (Sanazaro,  Goldstein,  Roberts,  and 
others,  1972).  Nonphysicians  began  to  be  involved  as 
measurement  became  more  sophisticated  and  the 
types  of  research  skills  needed  went  beyond  the 
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usual  expertise  of  physicians.  Measures  that  could 
be  used  by  external  reviewers  began  to  appear.  Dur- 
ing these  two  decades  there  were  major  advances  in 
quality  measurement  but  less  progress  in  QI. 

Now  we  are  at  the  beginnings  of  the  industrial 
era  in  QI.  AHCPR  is  already  funding  research  on 
outcomes  and  effectiveness,  as  well  as  development 
of  practice  guidelines,  review  criteria,  and  perfor- 
mance measures.  This  conference  was  designed  to 
help  set  AHCPR's  agenda  for  QI  research.  Mea- 
sures are  needed  both  for  more  detailed  work  on 
improving  systems  within  organizations  and  also 
for  large-scale  comparisons  among  organizations. 
Research  in  QI  is  expensive.  When  whole  organi- 
zations are  the  study  subjects,  the  scale  of  research 
becomes  very  large,  and  interdisciplinary  teams 
are  usually  needed.  The  research  is  complex, 
involving  clinical  data  bases,  computerized  review 
systems,  and  feedback  of  data  displays  to  influence 
physician  behavior. 


As  quality  of  care  becomes  a  competitive  advan- 
tage, new  sources  of  funding  are  emerging  for  QI 
research.  Some  large  health  care  plans  and  managed 
care  organizations  have  funded  their  own  internal 
departments  for  research  and  development  in 
quality  of  care.  Tools  for  utilization  review  and 
quality  measurement  are  sufficiently  in  demand 
that  investors  have  even  funded  some  research  and 
development  in  this  field.  This  creates  a  new 
dilemma,  however,  because  business-funded  mea- 
sures usually  are  not  in  the  public  domain.  Industry 
may  underinvest  in  this  research  because  the  pay- 
offs may  be  too  distant.  Government,  on  the  other 
hand,  has  insufficient  funds  for  some  of  the  larger 
efforts  needed.  Perhaps  the  solution  will  lie  in  the 
horizontal  form  of  keiretsu  (Kelly,  Port,  Tree,  and 
others,  1992) — that  is,  a  research  and  development 
consortium  including  government,  health  care 
plans,  hospital  chains,  managed-care  organizations, 
universities,  and  research  laboratories,  banded 


together  to  do  "precompetitive"  research  on  QI  in 
health  care  for  the  good  of  all. 
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Introduction 

Assuring  quality  in  health  care  delivery,  like 
assessing  the  quality  of  medical  treatment,  requires 
the  ability  to  assess  the  appropriateness  of  the  care 
and  the  quality  with  which  it  was  performed.  As 
daunting  as  these  tasks  are,  quality  assurance  (QA) 
attempts  to  go  beyond  them  in  order  to  change 
performance,  either  to  improve  it  or  to  prevent 
problems  from  arising.  Yet,  as  Leon  Wyszewianski 
(1988)  has  pointed  out,  the  majority  of  studies  have 
focused  on  the  methods  for  measuring  quality — the 
first  step — rather  than  the  means  for  achieving 
improvement. 

In  this  paper,  QA  refers  to  both  traditional  QA 
methods  already  in  use,  as  well  as  those  based  on 
continuous  quality  improvement  (CQI).  We  con- 
tinue to  talk  about  QA  but  agree  with  George 
Baker  (1992)  who  has  suggested  that  a  better  term 
would  be  quality  assistance — in  recognition  that 
none  of  the  techniques  actually  lives  up  to  the  con- 
cept of  assuring  quality  but  merely  assists  in  its 
attainment. 

This  discussion  draws  on  some  of  the  insights 
and  perspectives  of  organizational  theory  to  address 
basic  QA  issues,  particularly  how  to  use  data-based 
methods  to  improve  the  quality  and  delivery  of 
health  care.  I  will  maintain  that  the  boundaries 
between  successful  QA  models,  quality  improve- 
ment (QI),  and  utilization  review  (UR)  are  fuzzy, 
and  that  models  need  to  explicitly  recognize  their 
interrelationships  and  joint  effects  and  causes. 


Organizational  Models  and  Control 

In  academia,  recognizing  gradations  and  subtieties  is 
a  mark  of  a  thinking  person,  surely  a  scholar.  On 
careful  examination,  one  finds  few  things  that  are 
black  and  white.  Off  campus,  however,  the  world 
tends  to  think  in  terms  of  dichotomies  (Etzioni, 
1991,  p.  12). 

W.  Richard  Scott  (Scott,  1987;  Scott  and 
Backman,  1990)  describes  organizational  theory  in  a 
historical  context,  noting  the  basic  types  of  models 
and  arguments  that  have  been  dominant  during 
different  decades  and  the  impact  they  have  had  on 
the  issues  identified  and  solutions  proposed  by 
theorists,  practitioners,  and  policymakers. 

One  lesson  we  can  take  from  Scott's  work  is  to 
be  aware  of  the  limitations  on  our  thinking  and 
solutions  that  are  imposed  by  the  selection  of  basic 
models  of  the  problem.  Of  course,  these  limitations 
are  always  easier  to  see  with  hindsight.  For  exam- 
ple, in  the  early  1970s  organizational  models  of 
open  systems  and  ecological  descriptions  of  the  rise 
and  fall  of  organizations  helped  us  understand  the 
nature  and  impact  of  a  hospital's  interactions  with 
its  environment,  but  we  still  overlooked  the  com- 
plexity of  arrangements  between  hospitals.  Man- 
agers, on  the  other  hand,  were  busily  building  links 
and  connections  so  that,  when  the  American  Hospi- 
tal Association  first  included  questions  about  these 
links  in  its  annual  survey  during  the  1980s,  over  75 
percent  of  hospitals  reported  having  such  links 
already  in  place! 

Another  lesson  is  that  the  organizational  forms 
are  changing  and  evolving,  necessitating  that  models 
and  solutions  also  evolve — one  might  say  that  they 
need  CQI.  One  of  the  most  dramatic  and  pervasive 
examples  of  such  change  is  the  evolution  of  man- 
aged care  arrangements  in  health  care  (some  would 
argue  that  this  is  an  alias  for  some  forms  of  QA  and 
UR).  At  the  close  of  the  1970s,  prepaid  vs.  fee-for- 
service  modes  of  paying  for  care  were  associated 
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with  three  basic  organizational  types  (Luft,  1980). 
Group  and  staff  model  health  maintenance  organi- 
zations (HMOs)  treated  prepaid  patients  exclusively 
and  could  be  distinguished  primarily  by  type  of 
remuneration  and  employment.  The  third  model, 
the  individual  practice  association  (IPA),  covered 
most  situations  where  physicians  treated  both  fee- 
for-service  and  prepaid  patients.  In  addition  to  shar- 
ing this  characteristic,  IPAs  were  very  different  in 
organization;  they  were  generally  loose  conglomera- 
tions of  individuals  or  small  groups  of  physicians 
practicing  in  multiple  sites. 

Economists  like  Alain  Enthoven  (1980)  helped 
to  convince  a  generation  of  policymakers  that  pre- 
paid health  care  was  the  answer  to  concerns  about 
both  quality  assurance  and  cost-containment 
because  of  the  incentives  it  forced  into  provider 
decisionmaking  at  the  individual  and  group  levels. 

At  the  individual  level,  a  physician  could  make 
decisions  about  treatment  that  were  relatively  free 
of  influences  related  to  either  the  physician's  or 
patient's  pocketbook.  Meanwhile,  for  the  success  of 
the  group  of  providers  as  a  whole,  costs  had  to  be 
controlled  and  quality  maintained  to  continue  to 
attract  the  enrolled  population.  With  these  organi- 
zational level  incentives,  some  of  the  basic  hall- 
marks of  CQI — managing  the  processes  to  minimize 
costs  and  maximize  health — arose  first  and  most 
successfully  in  basic  prepaid  types  of  health  organi- 
zations. Those  managed  care  techniques  included 
gatekeeping,  review  of  expensive  decisions  to  treat 
prior  to  treatment,  discharge  planning,  and  shifts  to 
less  expensive  ways  to  provide  the  service  such  as 
same-day  surgery. 

During  the  1980s,  however,  some  of  these 
"truths"  about  the  fundamental  differences  in 
incentives  between  prepaid  and  fee-for-service  care 
evolved  along  with  the  more  complex  organiza- 
tional relationships.  By  the  end  of  the  decade,  not 
only  had  the  basic  incentive  structures  of  prepaid 
and  fee-for-service  care  evolved  into  myriad 
arrangements,  new  and  hybrid  organizational  forms 
associated  with  outpatient  care  had  developed,  irre- 
spective of  serving  prepaid  patients.  Managed  care 
techniques,  which  had  begun  with  prepaid  organiza- 
tions, spread  throughout  these  hybrid  organizations 
until  their  use  was  widespread  in  both  prepaid  and 
fee-for-service  patient  care. 

Indeed,  the  presence  of  one  of  the  basic  models 
formerly  associated  with  treating  prepaid  patients 
exclusively — for  example,  a  group  model  HMO 


— no  longer  identified  an  organization  as  treating 
prepaid  patients  most  of  the  time,  let  alone  exclu- 
sively. Prepayments  often  had  little  bearing  on  the 
incentive  structure  actually  experienced  by  the 
provider  responsible  for  making  health  care  deci- 
sions (Hillman,  1987;  Hillman,  Pauly,  and  Kerstein, 
1989).  Instead,  the  incentive  structure  actually  used 
by  an  organization  to  remunerate  its  providers 
can  turn  upside  down  the  incentives  supposedly 
inherent  in  prepaid  vs.  fee-for-service  care  (Flood, 
Fremont,  Goodrick,  and  others,  1991).  For  example, 
prepaid  practices  can  pay  physicians  on  the  basis  of 
services  delivered,  while  fee-for-service  practices 
can  pay  straight  salaries  to  physicians,  irrespective 
of  fees  generated. 

Yet  our  theory  and  public  policy  models,  includ- 
ing QA  models,  continue  to  assume  that  organiza- 
tions and  individual  providers  are  responding  to  a 
single  incentive  structure  that  is  determined  by  the 
simple  dichotomy  of  prepaid  or  fee-for-service  care. 
The  question  often  overlooked,  despite  the  com- 
plexity of  current  organizational  arrangements,  is: 
What  happens  in  an  organization  where  the  physi- 
cian/decisionmaker  has  mixed  incentives  or  multiple 
types  of  insured  patients?  What  happens  to  the  care 
of  the  rest  of  the  patients  when  managed  care  or 
QA  techniques  focus  only  on  a  subset  of  patients? 

In  a  recent  study  of  a  large  group  practice  with 
mixed  types  of  insured  patients,  my  colleagues  and 
I  tried  to  address  these  issues.  Preliminary  results 
suggested  that  doctors  believe  they  treat  all  their 
patients  the  same,  irrespective  of  managed  care  or 
insurance,  and  generally  argue  that  they  do  not 
differentiate  because  it  is  too  difficult,  as  well  as 
ethically  repugnant,  to  keep  track  of  insurance. 
However,  when  we  examined  their  behavior,  indi- 
cated by  rates  of  using  services,  we  found  that  they 
apparently  can  and  do  differentiate  among  classes 
of  patients  (Flood,  Freemont,  White,  and  others, 
1992).  For  example,  referrals  to  colleagues  in  the 
same  clinic  were  higher  among  prepaid  patients, 
while  rates  of  ancillary  services  were  lower.  When 
educational  programs  focusing  on  the  need  to 
reduce  the  use  of  services  (sponsored  by  the  prepaid 
plan  but  supposedly  not  targeted  to  this  group  of 
patients)  were  conducted  and  rates  of  service  use 
before  and  after  the  educational  effort  were  moni- 
tored, physicians  responded  most  strongly  in  areas 
that  did  not  require  a  significant  change  in  their 
practice  style.  For  example,  they  changed  their 
prescribing  habits  by  shifting  from  brand-name  to 
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generic  drugs  but  did  not  reduce  the  per  patient  rate 
of  prescriptions.  Palmer  (1988)  found  similar  results 
in  her  examination  of  feedback  to  physicians  in 
ambulatory  settings. 

The  third  lesson  from  Scott's  (Scott,  1987;  Scott 
and  Backman,  1990)  historical  review,  then,  is  the 
potential  for  gaining  new  insights  from  applying 
current  organizational  models  to  the  questions  of 
when  and  why  some  methods  of  QA  work  better 
than  others.  To  this  end,  I  will  review  the  evolution 
of  those  models  as  they  relate  to  methods  to  achieve 
successful  organizational  performance. 

Basic  Organizational  Model  and  Quality 
Assurance 

In  general,  organizational  forms  are  developed 
in  an  attempt  to  rationalize  the  process  by  which  an 
organization  achieves  its  objectives.  Since  the  1960s, 
many  have  argued  that  there  are  two  basic  types  of 
organizational  models:  the  bureaucratic  model, 
which  uses  centralized  control  and  authority  to 
organize  work,  and  the  professional  model,  which 
decentralizes  control  and  authority  to  the  worker 
and  depends  upon  professional  training  and  peer 
review  to  control  work.  Although  the  methods 
differ,  the  models  share  a  common  goal  to  reduce 
uncertainty  and  control  work,  either  in  procuring 
customers,  controlling  other  inputs,  organizing  work 
efficiently,  carrying  out  appropriate  activities,  assur- 
ing quality,  or  declaring  the  product  finished. 

Because  these  two  models  differ  so  fundamen- 
tally in  their  approach  to  controlling  and  evaluating 
work  processes,  many  theorists  in  the  1960s  argued 
that  they  were  inherently  incompatible.  Critics 
maintained  that  professionals  in  a  bureaucratic 
organization  would  be  dissatisfied,  and  their  work 
would  be  negatively  affected  by  bureaucratic  forms 
of  evaluation  and  control.  There  were  two  primary 
reasons  that  this  explanation  gave  way  to  more 
complex  models  of  work  by  professionals  in  health 
care  organizations.  First,  professionals  were  not  as 
distressed  or  hampered  in  carrying  out  their  work  as 
the  "fundamental  conflicts"  might  suggest.  Second, 
the  simple  dichotomy  of  bureaucrat  vs.  professional 
was  clearly  inadequate  to  describe  organizations  of 
any  type.  Models  of  organizational  theory  began  to 
concentrate  instead  on  examining: 

•   Multiple  dimensions  of  control  rather  than 
simplistic  dichotomies. 


•  The  extent  to  which  professional  values  and 
expertise  could  be  accommodated  within 
organizations. 

•  The  extent  to  which,  in  any  work  setting,  the 
right  way  to  organize  depends  on  the  task  to 
be  accomplished. 

Health  care  organizations  have  always  operated 
by  using  both  professional  and  bureaucratic  forms 
of  control.  Some  have  argued  that  there  were  two 
independent  lines  of  authority,  one  for  physicians 
and  the  other  for  everyone  else.  Georgopoulos  and 
Mann  (1962)  in  their  exposition  of  work  in  hospitals 
implied  that  these  two  lines  of  authority  acted  more 
like  a  lobotomized  brain  rather  than  an  integrated 
right  and  left  brain  in  coordinating  activities.  But 
others,  like  Scott  (1982),  described  three  models  for 
embedding  professionals  effectively  into  an  organi- 
zation: the  autonomous  model,  where  professionals 
retain  authority  to  control  and  evaluate  themselves 
as  a  group;  the  heteronomous  model,  where  profes- 
sionals are  subject  to  more  line-authority  control; 
and  the  departmentalized  model,  where  profession- 
als are  segregated  into  a  department  that  can  differ 
from  the  rest  of  the  organization.  Hospitals  actually 
have  all  three  forms:  medical  staff,  nursing  staff,  and 
clinical  laboratories,  for  example. 

Describing  a  modern  version  of  this  argument, 
Scott  and  Backman  (1990)  refer  to  the  interface  of 
the  bureaucratic  and  professional  models  in  hospi- 
tals as  a  "macromatrix"  where  work  is  accomplished 
by  teams,  some  of  whose  members  are  profession- 
als, and  where  different  aspects  of  work  are  subject 
to  professional  and  bureaucratic  types  of  control. 
[See  also  Burns  (1989)  for  a  description  of  matrix 
organizations  in  health  care  and  Scott  (1987), 
Shortell  (1983),  and  Alexander  and  Amburgey  (1987) 
for  discussions  of  the  "modern"  look  of  the  profes- 
sional-bureaucratic partnerships  in  health  care.] 

Wyszewianski,  Wheeler,  and  Donabedian  (1982) 
make  a  similar  point  when  they  distinguish  between 
two  types  of  efficiency  in  producing  health  care: 
production  efficiency,  requiring  efficiency  in  manag- 
ing inputs  and  outputs,  and  clinical  efficiency, 
requiring  professional  judgments  about  the  appro- 
priateness of  clinical  care  and  suitability  of  skills 
to  carry  it  out. 

These  ideas — that  control  of  work  can  be  done 
well  if  the  form  is  appropriate — suggest  a  relatively 
benevolent  view  of  the  basic  process  of  quality 
control:  it  is  a  way  to  learn  from  your  mistakes  and 
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to  perfect  the  production  of  care.  That  is,  problems 
in  the  production  of  work  are  due  to  the  system  and 
can  be  controlled  by  better  management  or  better 
strategic  planning.  Likewise,  problems  in  profes- 
sional decisionmaking  are  due  to  errors  that  are 
inadvertent.  To  prevent  such  errors,  QA  involves 
periodic  reviews  by  mentor-peers  who  can  under- 
stand the  vagaries  introduced  by  individual  cases 
and  help  to  identify  problems  that  can  be  corrected 
in  the  future.  This  type  of  model  of  control  assumes 
that  quality  can  be  assured  through  monitoring  and 
correcting  the  system  for  producing  health  care. 
Perhaps  these  methods  need  to  be  bolstered  at  the 
individual  practitioner  level  by  getting  rid  of  the 
few  really  "rotten  eggs"  but  mostly  by  rewarding 
good  performances  and  making  sure  bad  care  is 
embarrassing  enough  to  be  worth  avoiding.  Basi- 
cally, the  model  operates  on  the  carrot  side  of  a 
carrot-stick  continuum  and  blames  the  system 
rather  than  the  individual. 

These  types  of  assumptions  have  underpinned 
the  basic  approach  of  the  Joint  Commission  on 
Accreditation  of  Healthcare  Organizations 
(JCAHO)  during  the  1970s  and  the  beginning  of  the 
1980s.  Almost  all  hospitals  were  at  least  provision- 
ally accredited  by  JCAHO,  not  because  no  serious 
problems  were  found  but  because  the  Commission 
believed  that  every  organization  would  correct  the 
situation  when  given  the  "forced  opportunity"  to  do 
so.  The  Commission,  and  indeed  the  benevolent 
view  of  quality  control  in  health  care  production 
processes,  came  under  fire  in  the  1980s  for  "cover- 
ing up"  some  egregious  problems  and  for  failing  to 
ensure  that  reasonable  improvement  actually 
occurred  or  that  serious  problems  were  avoided. 

These  problems,  plus  widespread  disillusion- 
ment with  regulatory  peer  review  processes  such  as 
Medicare's  Professional  Standards  Review  Organi- 
zations (PSROs)  (Scott,  Bies,  and  LaPlante,  1980), 
gave  ascendancy  to  another,  less  benevolent  model 
of  health  care  production  processes.  The  definition 
of  QA  was  revised  to  reflect  a  system  designed  to 
protect  against  abuses,  with  the  expectation  that 
anyone  who  could  get  away  with  abusing  the  system 
for  personal  benefit  would  do  so. 

QA  as  it  evolved  in  the  1980s  had  some  schizo- 
phrenic assumptions  in  its  approach  as  formulated 
in  public  policy:  drop  those  namby-pamby  "carrot 
approaches"  of  the  1970s.  Instead  give  free  rein  to 
competition  to  solve  our  problems  in  both  efficiency 
and  quality  and,  at  the  same  time,  rein  in  abuse 


through  monitoring  everyone's  behavior  and  using 
sticks  such  as  withholding  payment  or  levying  fines 
when  abuse  is  uncovered. 

Before  exploring  these  next-generation  models, 
it  is  interesting  to  note  the  parallels  between  these 
earlier  models  of  QA  and  peer  review  and  the  cur- 
rent emphasis  on  CQI  in  health  care  (for  example, 
Berwick,  Godfrey,  and  Roessner,  1991;  Joint  Com- 
mission on  Accreditation  of  Healthcare  Organiza- 
tions, 1990).  To  wit,  CQI  involves  marrying  a  "hard" 
industrial  engineering  approach — that  is,  monitor- 
ing and  gathering  data  about  the  production 
process — to  a  "soft"  approach  featuring  an  organi- 
zation-wide QA  culture.  This  culture  should  view 
the  workers  as  basically  well-intentioned  and  the 
best  problem-solvers,  so  long  as  they  are  "pro- 
tected" from  being  blamed  by  administration  if  mis- 
takes are  found  and  "listened"  to  by  someone 
authorized  to  implement  their  solutions. 

The  Failure  of  "Benevolent" 
QA  Methods 

Three  basic  theories  have  been  put  forward  to 
explain  why  PSROs  and  "carrot  approaches"  to  QA 
failed  in  the  1970s  and  1980s:  the  agency  theory, 
rational  control  systems  in  organizations,  and  insti- 
tutional perspectives  and  rational  myths. 

The  Agency  Theory 

Americans  seem  to  have  a  hard  time  accepting  that 
the  problem  is  one  of  defects  in  the  system  rather 
than  of  individual  rascals  whom  they  can  throw 
out. . .  (Etzioni,  1991,  pp.  9-10). 

Agency  theory,  where  the  professional  acts  as 
the  informed  agent  to  make  decisions  for  the 
patient,  comes  out  of  neoclassical  economic  theories 
about  what  motivates  people  (self-interest)  juxta- 
posed with  what  the  professional  model  of  health 
care  demands  from  providers  (decisionmaking  that 
is  independent  of  financial  concerns).  Evidence  for 
the  failure  of  agency  models  due  to  self-interest  on 
the  part  of  the  agent  comes  from  several  types  of 
studies.  The  first  type  documents  evidence  of  extra- 
high  rates  of  utilization  associated  with  our  predom- 
inately fee-for-service  system  compared  with  those 
of  other  industrialized  nations  (McPherson,  Strong, 
Epstein,  and  Jones,  1981).  A  second  set  of  studies 
documents  services  reflecting  physician-induced 
demand  rather  than  demands  based  on  the  popula- 
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tion's  health  needs  (Rossiter  and  Wilensky,  1983; 
Wennberg,  1985;  Wennberg,  Barnes,  and  Zubkoff, 
1982).  The  third  category  examines  the  evidence  of 
higher  rates  of  utilization  associated  with  physician 
self-referral — that  is,  patients  are  referred  for  a  ser- 
vice that  in  turn  provides  the  referring  physician 
with  a  form  of  kickback  tied  to  receiving  referrals 
(Iglehart,  1991). 

The  solution:  put  into  place  QA  systems  that  will 
counteract  the  natural  tendency  for  providers  to  act 
in  their  own  behalf  instead  of  the  patient's  behalf 
(Dranove  and  White,  1987)  and  design  QA  systems 
to  counterbalance  the  self-interest  of  the  "sponsor" 
of  the  activity.  For  example,  if  the  review  is  spon- 
sored by  an  insurance  group,  including  public  insur- 
ance groups,  the  potential  conflict  of  self-interest  of 
the  insurance  company  needs  to  be  monitored;  but 
this  is  seldom  the  case. 

Should  QA  systems  have  QA  systems  to  moni- 
tor their  results?  Medicare's  Super-PRO  program 
would  appear  to  reflect  some  of  this  thinking, 
although  their  primary  purpose  is  still  to  serve  the 
interests  of  Medicare. 

Based  on  the  economic  model  of  quality  control, 
these  incentives  "inherent"  in  prepaid  insurance 
should  work  to  prevent  overuse  of  care,  but  not  nec- 
essarily to  prevent  its  underuse.1  If  these  incentives 
exist,  why  do  some  managed  care  plans  precertify 
use  of  services  and  use  concurrent  reviews  to  evalu- 
ate the  appropriateness  of  "extra"  days  in  the  hospi- 
tal rather  than  simply  monitor  for  underuse?  Is  it 
because  physicians  "forget"  their  self-interest?  This 
could  be  due  to  old  habits  learned  during  medical 
school  and  residency  training,  to  current  practice 
styles  established  when  they  or  their  colleagues 
treated  fee-for-service  patients,  or  to  pressures  from 
other  self-interested  parties.  On  the  other  hand,  it 
could  be  that  managed-care  insurers  are  acting  in 
their  own  self-interest,  irrespective  of  the  provider's 
interest.  As  noted  above,  in  modern  health  care 
organizations  remuneration  of  the  physician  or  hos- 
pital may  not  be  consistent  with  the  type  of  insur- 
ance. Instead  the  incentives  inherent  in  prepaid  care 
affect  the  self-interest  of  the  managed-care  plans 
themselves,  not  necessarily  the  providers. 


'Underuse  is  seldom  invoked  as  a  problem  in  American  health 
care,  except  when  fighting  off  national  health  insurance  and  the 
evils  of  rationing;  its  moral  and  economic  implications  deserve 
more  attention. 


Rational  Control  Systems  in  Organizations 

There  is  a  trade-off  curve  between  control  and  con- 
sensus: that  is,  for  any  given  level  of  activation,  the 
greater  the  consensus,  the  less  need  for  control. 
Which  mix  is  used  is,  of  course,  not  without  its  conse- 
quences; it  affects  the  level  of  alienation  and  of  resis- 
tance, and  hence  the  future  capacity  to  guide 
(Etzioni,  1991,  p.  39). 

The  second  explanation  is  rooted  in  rational 
models  of  organizations:  the  way  to  "control  work," 
if  professional  ethics  and  training  are  insufficient,  is 
to  hold  workers  accountable  by  making  their  work 
visible  to  others  whose  opinions  matter.  In  profes- 
sional work,  peer  review  is  needed  because  only 
peers  can  truly  judge  the  quality  of  one's  work.  This 
is  backed  up  by  a  potential  for  malpractice  litigation 
for  substandard  care  and,  to  a  limited  extent,  by 
financial  fines  and  professional  reprimands  for  poor 
performance. 

One  assumption  basic  to  this  approach  is  that 
internal  review  by  appropriate  members  of  the 
organization  is  more  powerful  than  an  external 
review  because  the  evaluation  will  be  seen  as  legiti- 
mate, authorized,  and  worth  caring  about.  In  other 
words,  one's  true  peers  are  those  who  understand 
the  situation,  both  in  terms  of  the  work  carried  out 
within  a  given  specialty  and  the  idiosyncratic 
requirements  of  treating  one's  specific  patients 
within  the  limitations  of  the  practice  setting.  In  con- 
trast, if  evaluations  and  QA  activities  are  seen  as 
external — that  is,  without  an  understanding  of  local 
variations  in  the  production  process  and  without 
concern  for  disruption  of  the  organization  caused  by 
the  QA  process  or  its  outcome — the  result  may  be  a 
systematic  attempt  by  providers  to  ignore,  game,  or 
bypass  QA  or  UR  whenever  possible. 

Another  assumption  basic  to  the  rational  model 
emphasizes  the  power  of  evaluations  that  have  con- 
sequences for  the  individual  being  evaluated.  In  this 
view,  people  prefer  to  receive  a  good  evaluation; 
therefore  workers  will  seek  to  improve  their  evalua- 
tion irrespective  of  whether  that  change  actually 
improves  the  quality  of  their  performance.  Ideally, 
of  course,  the  evaluations  are  accurate  and  appro- 
priate; if  not,  reactivity  to  the  performance  criteria 
can  result  in  an  appearance  of  improvement  in 
performance  rather  than  motivating  the  worker  to 
seek  real  QI.  Or,  as  W.  Richard  Scott  once  para- 
phrased an  old  song:  When  you're  not  near  the  goal 
you  love,  you  love  the  goal  you're  near. 
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Nonetheless,  there  is  some  evidence  that 
providers  will  respond  to  evaluation  by  peers  "for 
its  own  sake/'  For  example,  physicians  have 
responded  to  internally  imposed  peer  review.  Dyck, 
Murphy,  Murphy,  and  others  (1977)  found  that  rates 
of  healthy  tissue  removal  associated  with  appendec- 
tomy dropped,  without  any  need  to  reprimand 
physicians,  when  criteria  about  acceptable  rates 
were  made  explicit.  However,  physicians  knew  that 
peers  were  going  to  monitor  their  rates  in  the  future 
with  undetermined  consequences. 

In  another  study  of  interacting  drug  combina- 
tions, physicians  changed  their  overall  rates  of  order- 
ing any  prescription  drug  when  the  computerized 
system  for  monitoring  interacting  drug  combinations 
was  implemented  (Cohen,  Flood,  Himmelberger, 
and  others,  1980).  Wennberg,  Blowers,  Parker,  and 
Gittelsohn  (1977)  found  that  physicians  responded 
as  expected  to  external  feedback  that  their  rate  of 
hospitalization  for  a  particular  disease  was  unusually 
high,  without  any  compulsion  except  continued 
monitoring  of  performance. 

Finally,  the  work  by  the  Maine  Medical  Founda- 
tion provides  many  examples  of  how  a  study  group 
of  physicians,  when  given  feedback  on  utilization 
with  evidence  that  some  physicians  had  unusually 
high  rates,  resulted  in  a  reduction  of  the  outlier  rates 
over  time  (Keller,  Chapin,  and  Soule,  1990;  Keller, 
Soule,  Wennberg,  and  Hanley,  1990).  However, 
some  recent  evidence  suggests  that  physician  profiles 
showing  the  relative  rates  of  services  use,  when 
divorced  of  any  apparent  consequences,  appeared  to 
have  no  effect  on  behavior  (Wones,  1987). 

Overall,  the  basic  approach  to  QA  implicit  in 
this  model  is  neither  a  completely  self-serving  view 
of  actions  (like  the  agency  theory)  nor  a  completely 
benevolent  view  of  QA:  workers  will  do  the  right 
thing  if  they  think  their  work  is  being  seen  by  legiti- 
mate others,  but  otherwise,  they  may  not. 

One  reason  professionals  may  not  do  the  right 
thing  in  the  absence  of  evaluation  is  because  they 
think  they  know  what  to  do  (professionals  are 
trained  to  believe  that  they  know  what  the  right 
thing  is),  but  there  are  few  opportunities  to  know 
what  others  do  or  learn  the  comparative  advantage 
of  doing  something  differently.  While  continuing 
medical  education  and  reports  in  medical  journals 
are  supposed  to  fulfill  this  function,  the  level  of 
detail  is  seldom  sufficient  to  inform  practitioners 
how  they  might  improve  their  techniques  or  manage 
the  situation  more  effectively. 


Institutional  Perspectives  and 
Rational  Myths 

. . .  non-rational  behavior  is  natural;  rational  conduct  is  arti- 
ficial; it  must  be  manufactured ....  [and  therefore]  it  has  a 
cost  (Etzioni,  1991,  p.  68). 

Institutional  theory  adds  back  into  the  picture 
four  important  insights  that  tended  to  get  over- 
looked in  recent  QA  and  QI  work: 

1.  Some  organizational  behavior  involves  rational- 
izing action  rather  than  rational  action. 

2.  Organizations,  as  well  as  individuals,  will 
respond  to  how  they  are  evaluated,  regardless  of 
whether  it  improves  their  process  or  product. 

3.  Organizations  can  influence  and  be  influenced 
by  the  environment. 

4.  Government  is  not  a  monolithic  substitute  for 
society;  it  is  a  set  of  competing  institutions  that 
have  their  own  goals  and  accountability. 

Meyer  and  Scott  (1983)  and  Meyer,  Scott,  and 
Deal  (1981)  argued  that  it  is  useful  to  distinguish 
between  two  facets  of  an  organization's  environ- 
ment: the  technical  and  the  institutional.  Technical 
aspects  of  the  environment  emphasize  the  efficiency 
and  quality  of  the  product  and  reward  organizations 
on  the  basis  of  how  well  their  products  measure  up 
to  these  standards.  Institutional  environments,  on 
the  other  hand,  generally  operate  when  these  crite- 
ria are  unknown  or  difficult  to  ascertain;  instead 
they  resort  to  evaluating  the  process  by  which  the 
product  is  produced.  Were  the  rules  and  standards 
followed?  Did  they  do  as  well  as  their  peers,  organi- 
zationally or  individually?  Did  they  have  approved 
structures  in  place  to  ensure  good  products? 

Scott  and  several  others  (see  Mick,  1990)  have 
argued  that  the  health  care  system  is  strongly  influ- 
enced by  both  kinds  of  environments.  That  is,  tech- 
nical capacity  and  the  ability  to  compete  efficiently, 
through  vertical  or  horizontal  integration,  and  effi- 
cient production  processes  influence  some  aspects  of 
organizational  effectiveness.  Other  aspects  of  the 
environment  are  irrational  in  the  sense  that  health 
care  organizations  must  respond  to  them  regardless 
of  their  import  for  quality  or  efficiency.  Require- 
ments for  accreditation  can  be  an  example  of  this 
behavior,  including  those  that  require  the  form  if  not 
the  function  of  QA  or  CQI  systems  to  be  in  place. 
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More  broadly,  licensing,  accreditation,  tort  law, 
and  regulations  associated  with  quality  control 
abound  in  health  care.  Even  when  they  are  based  on 
widely  held  beliefs — what  Meyer  and  Rowan  (1977) 
have  labeled  "rational  myths" — rather  than  on  evi- 
dence of  their  import  for  quality,  they  have  an  abil- 
ity to  influence  organizational  performance  regard- 
less of  the  validity  of  the  claim. 

These  then  are  the  first  two  insights  from  an 
institutional  perspective  of  organizations: 

1.  QA  can  have  very  symbolic  aspects  for  organiza- 
tions as  well  as  individual  practitioners,  particu- 
larly when  its  outcome  affects  the  accreditation 
status  or  reimbursement  for  care  for  a  health 
care  organization.  The  old  view  of  control  was 
that  internal  review  and  evaluation  are  always 
better;  the  new  view  is  to  recognize  that  the  most 
powerful  and  least  obtrusive  form  of  control  is 
through  beliefs  (Scott  and  Backman,  1990). 

2.  The  process  of  QA — or  more  public  processes 
such  as  HCFA's  releases  of  hospital  death 
rates — can  cause  organizations  to  respond  in 
ways  that  may  not  produce  any  real  improve- 
ment in  quality  and  may  detract  from  their  will- 
ingness to  conduct  difficult  investigations  of 
problems  in  quality. 

Some  proponents  of  organizational  environments 
would  have  us  believe  that  organizations  are  hapless 
victims  of  their  environments  or  at  best  can  merely 
react  (or  proact)  to  offset  the  influence  of  environ- 
ments. Others  point  out  the  extent  to  which  health 
care  practitioners  and  health  care  providers  can 
influence  the  shape  of  their  regulatory  environment 
and  also  their  competitive  market.  Certainly  a  host 
of  lobbyists  in  State  and  Federal  governments  would 
be  surprised  to  learn  they  had  no  effect  on  the 
development  or  administration  of  laws  and  budgets 
affecting  health  care.  Indeed,  one  of  the  maxims  of  a 
determined  lobbyist  is  Yogi  Berra's  line:  "It  ain't 
over  till  it's  over."  Even  after  a  law  is  passed,  it  is 
possible  to  affect  its  impact  by  influencing  how  it  is 
interpreted  by  those  administering  the  regulations. 

The  last  two  insights  from  the  institutional  per- 
spective are: 

3.  The  path  between  health  care  organizations  and 
their  environments  is  two-way.  One  of  the  most 
important  "battlefields"  for  both  sides  is  the 
struggle  to  define  QA:  What  should  be 
evaluated?  Who  should  evaluate?  With  what 
consequences?  Whose  beliefs  should  dominate? 


4.  Government  (one  aspect  of  the  health  care 
organization's  environment)  is  not  a  monolithic 
institution  serving  as  society.  Instead  it  is  a  multi- 
tiered,  loosely  connected  set  of  agencies  and 
fragile  coalitions  with  their  own  obligations  and 
domains  that  may  be  in  conflict  with  each  other 
or  at  least  stand  in  counterpoint  to  each  other. 

So  how  should  QA  be  organized  to  actually 
assure  quality? 

Promises,  Promises:  The  Future  for 
Quality  Control 

That,  at  bottom,  is  what  this  is  really  all  about:  recap- 
turing hope.  We  think  there  is  at  least  one  way  out  for 
American  health  care  in  the  1990s,  namely,  for  health 
care  organizations  to  learn  and  to  apply  the  methods 
of  modern  quality  management  as  their  primary 
operating  strategy  (Berwick,  Godfrey,  and  Roessner, 
1990,  p.  xvi). 

Continuous  quality  improvement  is  the  promise 
put  forward  for  the  1990s  to  address  the  problems — 
real  and  imagined — in  QA  in  American  health  care 
delivery.  CQI  is  often  used  interchangeably  with 
terms  such  as  total  quality  management,  industrial 
quality  control,  and  the  Deming  approach.  Some- 
times proponents  are  picking  up  only  part  of  the 
message,  such  as  data  gathering  and  problem  solv- 
ing by  the  interested  parties.  But  the  package,  to  be 
successful,  requires  implementation  of  all  features. 

Its  proponents  include  an  amalgam  of 
researchers,  industrial  engineers,  accrediting  groups, 
practitioners,  and  industrial  quality  control  experts. 
The  approach  requires  several  basic  features: 

•  Define  the  quality  problem  in  such  a  way  that  it 
can  be  addressed  with  systematically  gathered 
data. 

•  Set  up  the  processes — defining  the  problem, 
gathering  the  data,  and  proposing  the  solutions 
— so  the  right  people  are  included.  The  right 
people  are  those  whose  production  is  being 
examined  and  those  with  the  clout  to  carry  off 
any  proposed  changes. 

•  Create  a  new  kind  of  corporate  culture  about 
solving  quality  problems,  which  needs  to 
pervade  all  levels  of  the  organization;  mistakes 
occur  because  of  system  problems,  not  because 
of  "bad"  workers.  Everyone  benefits  from  better 
production  processes,  even  if  only  in  terms  of 
morale  and  pride. 
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The  philosophical  approach  of  CQI  is  similar  in 
some  ways  to  that  of  groups  like  the  Maine  Medical 
Foundation  (MMF).  Both  start  with  the  premise 
that  wide  variation  in  practice  indicates  that  some- 
thing is  amiss — not  all  rates  can  be  right,  even  if  we 
do  not  know  which  rate  is  right — but  the  two  groups 
part  company  in  their  philosophical  approach  to 
which  rate  is  right.  The  MMF  approach  targets  the 
outliers  (usually  high  utilization)  with  the  view  that 
outlying  providers  should  alter  their  performance  to 
look  more  like  the  typical  performance  of  their 
peers.  The  CQI  approach,  in  contrast,  argues  that 
the  outliers  on  the  side  of  good  quality  should 
become  the  benchmark  against  which  everyone 
should  strive:  to  try  to  be  the  best,  not  typical.  The 
differences  ostensibly  are  due  to  stressing  costly  uti- 
lization in  the  MMF  and  excellent  quality  in  the 
CQI.  But  the  importance  of  this  distinction  tends  to 
get  overlooked  by  some  proponents,  with  the  result 
that  better  quality  gets  mixed  up  with  efficiency  of 
production.  As  Wyszewianski  and  his  colleagues 
(1982)  discuss  at  length,  there  is  a  difference 
between  maximizing  quality  in  the  production 
process  and  in  clinical  outcomes. 

Although  the  paradigm  of  CQI  is  new  to  the 
health  industry,  several  efforts  are  underway.  In 
1987  a  National  Demonstration  Project  on  Quality 
Improvement  in  Health  Care  was  launched.  A  team 
from  21  health  care  organizations  was  paired  with 
an  industrial  quality  expert  in  an  attempt  to  use 
CQI  methods  to  address  major  problems  in  their 
organizations.  Their  experiences  are  reported  by 
Berwick,  Godfrey,  and  Roessner  (1990).  Although  it 
is  a  story  basically  ending  with  successes,  most  of 
the  problems  tackled  were  far  removed  from  clinical 
issues  in  the  production  of  health  or  areas  typically 
of  concern  to  QA  efforts. 

Others,  including  some  of  the  21  organizations, 
are  trying  to  tackle  clinical  production  processes. 
For  example,  Carol  Sox  of  Dartmouth  Medical 
School  is  experimenting  with  industrial  engineering 
models  of  cardiac  surgery  as  a  production  process. 
Instead  of  defining  the  product  as  a  health  care  ser- 
vice (like  hospitalization  or  surgery),  the  product  is 
health.  Thus,  death  within  a  specified  period  of  time 
is  the  "nonconforming  product."  After  adjusting  for 
case  mix,  she  modeled  the  process  using  typical 
quality  control  assumptions.  Without  having  to 
solve  the  "which  level  is  right"  problem,  quality 
control  charts  simply  plot  standardized  mortality 
over  time.  The  team  of  workers  then  must  judge 


when  points  appear  to  be  sufficiently  out  of  bounds 
to  investigate  for  potential  problems  needing 
solution  or  for  indications  of  extra-good  perform- 
ance that  could  become  a  new  benchmark  for 
other  providers. 

The  promise  of  CQI  to  be  able  to  address  these 
more  difficult  issues  exists,  but  CQI  is  not  a  magic 
solution.  The  most  difficult  issues  are:  How  do  you 
get  people  to  ask  the  right  questions?  How  do  you 
get  people  to  want  to,  and  know  how  to,  correct 
deficiencies?  Are  the  production  processes  involved 
in  producing  good  health  services  the  same  as  those 
that  produce  good  health? 

Iconoclastic  Challenges  to  Inform  and 
Reform  QA  Policies 

Far  from  being  highly  subject  to  design,  far  from 
being  highly  malleable,  societies  (and  sub-societies) 
typically  are  highly  resistant  to  change.  The  extent  to 
which  societies  are  malleable  depends  to  a  significant 
extent  not  merely  or  even  firstly  on  the  attributes  of 
the  controlling  parts,  but  significantly  on  those  of  the 
guided  units,  and  their  relationship  to  the  controlling 
overlai  (Etzioni,  1991,  p.  40). 

Open  the  black  box  of  organizations.  It  is  one 
thing  to  design  public  policy  for  QA  at  a  macro- 
level  in  order  to  avoid  the  pitfalls  of  micromanage- 
ment  (a  valuable  goal),  but  it  is  quite  another 
when  such  policy  reflects  our  ignorance.  We  need 
more  research  regarding  several  underlying 
questions  about  the  promise  of  CQI.  For  example, 
what  are  the  circumstances  when  QA  is  effective  in 
achieving  desirable  change?  What  are  the  negative 
consequences  that  even  "good"  QA  can  have? 

Examine  fundamental  assumptions  built  into 
QA  management  and  public  policies.  The  effective- 
ness of  CQI  in  health  care  appears  to  rest  on  two 
types  of  assumptions.  The  first  assumes  that  those 
CQI  methods  successfully  applied  to  other  indus- 
tries also  can  be  applied  to  health.  This  assumption 
often  takes  on  a  level  of  faith  that  cannot  be  ques- 
tioned. At  some  level  of  belief  this  assumption  is 
probably  necessary  to  get  past  the  naysayers  who 
would  dismiss  CQI  without  trial.  However,  it  is 
important  to  go  beyond  this  assumption  and 
evaluate  the  circumstances  when  CQI  can  be 
applied  successfully  and  when  not. 

The  second  type  of  assumptions  equate  CQI 
with  all  functions  of  QA  in  health.  In  particular, 
regulatory  bodies  charged  with  oversight  of  multiple 
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organizations  are  applying  CQI  methods  without 
recognizing  that  their  methods  contradict  the  sup- 
posed wisdom  of  CQI  itself:  it  will  not  work  if  you 
punish  people  on  the  basis  of  what  you  find — for 
example,  withholding  accreditation  or  payment 
— or  if  the  group  does  not  see  itself  as  a  team — for 
example,  hospitals  under  the  jurisdiction  of  a  Peer 
Review  Organization  (PRO).  Similarly,  isolated 
teams  in  several  organizations  are  trying  to  apply 
CQI  without  the  cooperation  or  involvement  of 
other  members  at  the  host  institutions.  This  too 
contradicts  some  of  the  basic  tenets  of  CQI  to 
involve  the  entire  organization  and  cross  all  aspects 
of  production. 

Some  assumptions  should  be  seriously  examined, 
not  taken  for  granted: 

•  The  behavior  of  practitioners,  but  not  organiza- 
tions or  government  agencies,  is  motivated  by 
self-interest. 

•  Any  form  of  organizing  prepaid  care  has  the 
same  set  of  incentives  to  motivate  behavior. 

•  Requiring  implementation  of  CQI  through 
regulatory  oversight  from  PROs  or  accreditation 
by  JCAHO  will  have  the  same  effect  as  building 
CQI  from  internal  pressures. 

•  CQI  can  occur  effectively  in  a  context  other 
than  maintenance  and  survival  of  a  particular 
organization  or  system  of  organizations.  New 
organizational  forms  and  new  health  policies, 
like  physician  payment  reform — maybe  even 
national  health  insurance  in  some  guise — present 
new  opportunities  as  well  as  new  challenges  to 
design  ways  to  assure  and  improve  quality. 
Research  on  QA  addressing  these  larger  issues 
and  contexts,  rather  than  focusing  exclusively 
on  the  narrow  issue  of  CQI,  is  needed. 

Practice  guidelines  may  present  the  biggest  QA 
challenge  of  all.  How  much  do  we  know  about  the 
symbolic  responses  to  their  implementation?  Or  the 
ways  to  convince  people  to  change?  Or  even  the 
extent  to  which  the  guidelines  are  based  on  real 
information  instead  of  rational  myths?  Are  we  bet- 
ter off  if  we  induce  all  of  our  hospitals  to  offer  "the 
best  care  in  the  world"  and  start  a  frenzy  of  organi- 
zations trying  to  imitate  the  "best  institutions" 
because  we  have  no  other  way  to  judge  whether  a 
hospital  is  good  except  it  "looks"  like  the  good  hos- 
pital? Is  there  even  such  a  thing  as  a  "good"  hospi- 
tal? Or  are  they  good  at  some  care  and  not  others? 


How  do  you  assure  quality  or  improve  the  process 
when  it  crosses  institutions?  What  do  you  do  if  you 
find  out  that  quality  is  poor  and  change  is  unlikely? 

CQI  challenges  the  value  of  conducting  exter- 
nally imposed  QA  as  well  as  using  traditional  forms 
of  internal  QA  such  as  peer  review.  Is  CQI  really 
the  most  effective  way  to  motivate  providers  to  pro- 
duce better  and  cheaper  health  care?  Is  it  thwarted 
by  the  presence  of  traditional  QA  or  oversight  by 
regulators  or  accrediting  bodies,  even  if  the  regula- 
tors and  accreditors  intend  to  implement  CQI  at  an 
extra-organizational  level? 

Asking  the  right  questions  is  the  most  important 
step  in  CQI.  In  this  context,  what  is  the  role  of  indi- 
vidual patient  preferences  for  outcomes  or  for  risk- 
taking  or  even  for  expending  resources  to  achieve 
quality?  How  do  you  use  CQI  to  examine  questions 
about  who  didn't  get  care  but  should?  Or  examine 
who  got  care  that  was  more  likely  to  do  harm  than 
good?  And  how  do  you  examine  the  quality  of  pre- 
vention of  health  problems? 

Pleasing  the  consumer  is  central  to  CQI.  For 
some  health  services,  like  immunizations  against 
epidemics  or  providing  care  for  the  poor,  society  is 
one  of  the  consumers.  How  well  can  you  address  a 
population's  interests  as  the  consumer,  and  how  do 
you  deal  with  consumers  who  need  care  but  cannot 
pay  for  it? 

Research  is  needed  to  fully  understand  if  and 
how  CQI  does  a  better  job  at  promoting  quality.  Is  it 
effective,  especially  considering  the  expenditure  of 
resources  to  carry  out  CQI?  Is  it  really  a  "package" 
approach  or  are  some  dimensions  more  necessary 
than  others?  Does  it  only  work  in  some  contexts?  Is 
a  supportive  corporate  culture  helpful  or  necessary? 
If  necessary,  how  do  you  promote  it?  For  example, 
should  the  methods  and  culture  of  QA  be  included 
in  medical  and  nursing  curricula? 

Conclusion 

We  are  attempting  exciting  and  stimulating  new 
ways  to  improve  the  quality  of  health  care.  CQI 
holds  much  promise,  not  only  because  it  has  the 
potential  to  improve  the  quality  of  health  care  ser- 
vices but  also  because  it  offers  the  opportunity  to 
broadly  reexamine  our  current  policies  and  methods 
and  to  do  so  in  a  way  that  engages,  rather  than  turns 
off,  the  producers  and  customers.  Much  work  needs 
to  be  done  to  carry  off  these  promises  and  to  find 
the  limitations.  If  we  attend  to  the  lessons  already 
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learned  from  previous  attempts  to  assure  quality 
and  from  organizational  theory,  perhaps  that  more 
modest  goal  of  assisting  quality  can  be  achieved. 
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Introduction 

The  increase  in  "value-driven"  competition 
among  health  care  providers  and  insurers,  coupled 
with  renewed  pressure  on  third-party  payers  to 
ensure  that  cost-containment  initiatives  do  not  com- 
promise quality  of  care,  has  increased  the  demand 
for  information  on  the  quality  and  efficiency  of  care 
delivered  in  all  settings.  The  demand  for  usable 
information  is  likely  to  escalate  with  the  continued 
diffusion  of  quality  management  techniques  in  the 
health  care  industry.  The  operative  term  is  "usable 
information."  Most  health  care  managers  and  clini- 
cians are  drowning  in  data,  but  data  are  not  neces- 
sarily information  and  not  all  information  is  usable. 

Usable  information,  in  the  context  of  both  qual- 
ity assurance  (QA)  and  quality  improvement  (QI) 
activities,  is  timely,  reliable,  and  actionable.  It 
empowers  managers  and  clinicians  to:  fix  what  goes 
wrong;  learn  from  what  goes  right;  reach  increasingly 
higher  levels  of  service,  efficiency,  and  technical 
quality;  achieve  better  outcomes  for  their  "cus- 
tomers"; and  know  when  they  have  done  so. 

Automated  information  systems  and  technolo- 
gies can  be  powerful  tools  for  producing  usable 
information,  but  are  we  using  these  systems  to  their 
full  potential?  This  paper  addresses  the  following 
questions: 

•  How  do  QA  and  QI  differ  with  respect  to  their 
information  needs? 

•  To  what  extent  can  currently  available  informa- 
tion systems  provide  potentially  usable  informa- 
tion for  QA  and  QI?  What  types  of  information 
are  being  used  now,  and  how  are  automated 
information  systems  being  used  to  provide  that 
information? 

•  What  additional  types  of  data  are  needed 
through  automated  information  systems  if 
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QA/QI  needs  for  usable  information  are  to  be 
met  in  a  low-cost,  timely,  and  reliable  manner? 

•  To  what  extent  are  information  technologies 
being  used  to  access,  analyze,  and  disseminate 
information  from  single  or  multiple  linked  infor- 
mation systems?  What  industry  practices  facili- 
tate or  impede  such  efforts? 

•  How  can  emerging  information  technologies  be 
used  to  increase  the  range  and  quality  of  data 
available  through  automated  information  sys- 
tems and  the  options  available  for  data  linkage, 
processing,  analysis,  display,  and  transmission? 
What  new  technologies  are  needed? 

•  What  are  the  pragmatic  and  policy  issues  that 
must  be  addressed  to  realize  the  potential  of  cur- 
rent and  emerging  information  technologies  to 
improve  health  care  quality  and  efficiency? 

Automated  Information  Systems  and 
Quality  Assurance/Quality  Improvement 

QA  has  been  described  by  Steinwachs  as  involv- 
ing the  measurement  of  quality  as  one  step  in  a 
process  that  "monitors  quality  of  care  indicators; 
conducts  quality  assessments  where  indicators  sug- 
gest potential  problems  may  exist;  makes  recom- 
mendations for  improving  care  based  on  the  assess- 
ment; and  continues  to  monitor  the  problems  into 
the  future"  (Shapiro,  Weiner,  and  Steinwachs,  1989). 

QI  may  be  described  as  the  application  of  scien- 
tific methods  to  understand  and  continuously 
improve  the  ability  of  all  processes  to  meet  the 
needs  of  customers.  The  principal  steps  in  the  QI 
process  include: 

•  Identifying  an  opportunity  for  improvement, 
typically  resulting  from  the  measurement  of 
defects  in  the  output  of  a  process  or  inefficien- 
cies in  the  process  itself  (for  example,  cost, 
cycle  time). 
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•  Diagnosing  the  root  causes  of  defects  in 
products  or  services  (outputs)  or  inefficiencies  in 
processes  and  differentiating  systemic  from 
extra-systemic  causes  (Kritchevesky  and 
Simmons,  1991). 

•  Exploring  alternative  interventions  to  achieve 
improvements. 

•  Implementing  solutions. 

•  Monitoring  performance  to  hold  the  gains. 

On  the  surface,  the  QA  and  QI  process  descrip- 
tions appear  very  similar,  but  in  implementation, 
they  generally  have  been  quite  different.  This  is 
particularly  true  in  hospitals,  where  the  majority  of 
QA  activities  have  been  mandated  and  where  QA 
has  typically  been  translated  into  "peer  review." 


Several  authors  have  described  the  differences 
between  QA  and  QI  (Andrews,  1991;  Berwick, 
1990;  Fainter,  1991;  Laffel  and  Blumenthal,  1989), 
which  are  summarized  in  the  Table.  These  differ- 
ences in  customers,  goals,  focus,  objectives,  meas- 
ures, actions,  and  other  features  all  contribute  to 
differing  information  needs. 

Information  Needs  for  QI/QA: 
Similarities  and  Differences 

In  health  care,  QI  encompasses  both  clinical  and 
operational  processes  and  includes  as  customers  all 
of  the  following:  the  "users"  (patients  and  their 
families),  "nonusers"  (the  population  for  whom  an 
insurer  or  a  health  care  delivery  organization 
assumes  responsibility),  public  and  private  payers  of 


Table.  Quality  assurance  vs.  quality  improvement  in  health  care  institutions 


Characteristic 

Customers 


Goal 
Direction 

Focus 

Performance 
measure 

Objective 

Defects  studied 

Viewpoint 
Participants 
Functions  involved 

Tampering 
Review  method 
Action  taken 

Measurement  and  reporting 


Quality  Assurance 

Regulators 


Regulatory  compliance 

Central  coordinator  or 
committee 

Physician 

External  standards 

Control 

Outliers/special  causes 
("extra-systemic") 

Reactive 

Peers 

Few — mainly  physician 

Common 
Summary 

Recommend  improvements 
Ad  hoc,  sporadic 


Quality  Improvement 

External:  patients,  enrollees, 
families,  payers,  etc.  Internal: 
caregivers,  managers, 
technicians,  support  staff,  etc. 

Meet  customer  needs 

Decentralized  through 
the  management  line 

Processes 

Capability/need 

Breakthrough 

Common  ("systemic")  and 
special  causes 

Proactive 

All 

Many —  clinician 
and  support  system 

Rare 

Analysis 

Implement  improvements 
Routine,  periodic 


Source:  Adapted  from  Berwick  (1990)  and  Fainter  (1991) 
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services  or  insurance  benefits,  and  all  of  the  clinical 
and  nonclinical  staff  involved  in  direct  care  or 
operational  support  processes. 

While  QA  programs  have  relied  on  data  to  iden- 
tify problems  in  the  quality  of  care  provided  by 
health  care  services,  historically  they  have  ignored 
the  issue  of  underutilization  in  their  service  area, 
enrolled  population,  or  patient  base.  Often  this  has 
resulted  from  a  lack  of  data  on  the  nonutilizing  pop- 
ulation. This  is  particularly  true  among  fee-for-ser- 
vice  providers  who  have  difficulty  updating  and 
maintaining  patient  files,  given  the  migration  of 
patients  among  providers  and  geographic  regions. 
For  products  or  services  that  are  population  based, 
such  as  screening  or  immunization  rates,  effective 
QI  efforts  will  require  information  on  both  users 
and  nonusers  of  a  provider's  services. 

Although  QA  activities  generally  have  been 
undertaken  primarily  to  comply  with  regulatory 
requirements,  the  goal  of  QI  is  to  meet  customer 
needs  more  effectively.  Data  on  the  needs  of  both 
external  customers  (enrollees  and  patients)  and 
internal  customers  (staff  working  within  the  care 
process  who  must  depend  on  other  staff  to  carry  out 
their  part  of  the  process)  are  crucial  to  QI  analyses. 
QA  departments,  especially  in  hospitals,  have 
tended  to  see  the  collection  and  use  of  patient  pref- 
erence or  satisfaction  data  as  more  in  the  domain  of 
planners  or  market  researchers  than  QA  staff  and 
the  use  of  staff  satisfaction  survey  data  as  principally 
a  human  resource  department  function.  QI  teams, 
however,  recognize  these  data  as  important  sources 
of  information  on  unmet  needs  and/or  deficiencies 
in  care  processes. 

QA  activities  traditionally  have  focused  on 
physician  decisions  and  actions  and  ultimately  on 
identifying  the  poor  performers  and  improving  or 
removing  them.  QI  philosophy  recognizes  that  peo- 
ple work  within  processes  and  that  their  ability  to 
work  up  to  their  capabilities  can  be  limited  by  the 
constraints  of  those  processes  (Berwick,  1989). 
Thus,  data  on  process  capability  and  performance 
are  a  requirement  for  QI  analyses. 

While  QI  does  identify  individual  outliers  in 
performance,  that  is  not  its  principal  focus.  The 
improvement  or  removal  of  statistical  outliers  yields 
lesser  gains  than  does  improving  the  process  capa- 
bility for  all  performers.  Thus,  the  central  focus  of 
QI  is  on  achieving  breakthroughs  in  process 
performance,  rather  than  identifying  "bad  apples." 
To  improve  a  process,  QI  efforts  depend  not  only 


on  the  outcome  (or  proximate  outcome)  measures 
typical  of  QA  indicators,  but  also  on  the  availability 
of  detailed  data  about  the  inputs  to  the  process  and 
the  steps,  characteristics,  and  temporal  relationships 
of  the  process  itself. 

QA  activities  tend  to  be  centralized  and 
removed  from  the  day-to-day  activities  of  health 
care  organizations,  while  QI  activities  engage  the 
efforts  of  all  members  of  the  organization  to 
improve  their  products  and  services  by  improving 
the  processes  in  which  they  work.  The  QI  process 
includes  the  implementation  of  solutions  to  process 
problems  by  those  who  work  in  and  manage  the 
process.  QA  managers,  on  the  other  hand,  may  rec- 
ommend solutions  only,  and  many  such  recommen- 
dations are  never  implemented.  Data  on  the 
chronology  of  QI  process  initiatives  must  be  docu- 
mented to  ensure  consistent  data  on  process 
performance  measures  (such  as  defect  rates  in  prod- 
ucts or  services,  cycle  time,  and  cost  per  cycle)  that 
will  be  used  to  assess  their  effectiveness  over  time — 
a  quality  control  function. 

In  summary,  QA  programs  generally  have 
focused  on  what  went  wrong  and  who  was  responsi- 
ble, not  on  why  things  went  wrong.  Nor  have  they 
used  benchmarking  techniques  to  examine  why 
those  same  things  go  right  in  another  population, 
delivery  setting,  or  care  context.  Providing  those 
responsible  for  the  quality  of  care  with  information 
only  on  what  went  wrong  often  leaves  them  without 
a  clue  as  to  how  care  might  be  improved.  It  is  the 
answers  to  the  "why"  questions  that  can  provide 
managers  and  clinicians  with  the  information  they 
need  to  improve  services  and  care  outcomes. 

QI  focuses  on  the  why  questions  by  systemati- 
cally analyzing  characteristics  of  the  processes  that 
resulted  in  the  observed  defects  in  care.  By  under- 
standing why,  how,  and  when  these  processes  fail, 
managers  can  act  to  redesign  the  processes  and 
thereby  improve  care  for  all  affected  patients. 

Recent  changes  in  the  focus  of  some  of  the  regu- 
latory agencies,  such  as  the  Joint  Commission  on 
Accreditation  of  Healthcare  Organization's  Agenda 
for  Change,  may  bring  the  QA  and  QI  processes 
closer  together.  In  any  event,  many  of  the  defects 
identified  through  QA  monitoring  can  provide  grist 
for  the  QI  process,  and  many  of  the  quality  meas- 
urement techniques  developed  for  QA  can  be  used 
in  a  QI  model.  The  data  required  to  support  the  QI 
process,  however,  are  both  broader  in  scope  and 
deeper  in  detail  than  those  used  for  QA. 
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Usefulness  of  Automated  Information 
Systems 

To  answer  questions  about  the  quality  and  effi- 
ciency of  health  care  delivery,  accurate,  reliable  data 
are  needed.  Changes  in  technology  and  medical 
practices  require  that  the  data  be  relatively  current 
(in  some  cases,  concurrent)  with  care.  In  addition, 
the  range  of  data  collected  must  be  expanded  to 
determine  whether  differences  in  quality  are  due  to 
differences  in  the  population  or  the  care  process  and 
to  understand  the  causes  of  known  quality  problems. 

A  basic  tool  of  quality  management  is  "bench- 
marking." Benchmarking  involves  seeking  out 
successful  innovations  in  other  settings  and  imple- 
menting them  in  one's  own  setting  when  appropri- 
ate, as  well  as  identifying  the  top  performers  in 
each  process  area  and  striving  to  exceed  their 
performance  levels. 

There  is  increasing  interest  in  benchmarking 
among  institutions,  health  care  plans,  and  delivery 
settings  (Chassin,  1991;  Siu,  McGlynn,  Morgenstern, 
and  others,  1991).  For  benchmarking  to  be 
effective,  however,  greater  uniformity  of  coding 
practices,  improved  consistency  in  data  definitions, 
and  greater  comparability  in  performance  measures 
are  necessary. 

These  factors,  plus  the  escalating  demand  for 
data  both  from  within  and  outside  health  care 
delivery  settings,  add  urgency  to  calls  for  further 
development  and  implementation  of  accurate, 
efficient,  and  low-cost  automated  data  collection, 
processing,  and  analysis  techniques. 

While  automated  information  systems  have 
been  used  extensively  in  health  care  to  support  QA 
monitoring  and — in  some  settings — interventions, 
only  recently  have  they  been  used  to  support  formal 
QI  initiatives. 

Use  of  Automated  Information  Systems 
to  Support  QA 

Traditional  QA  activities  may  be  divided  into 
two  types:  those  that  are  conducted  within  an  insti- 
tution or  health  plan,  usually  to  monitor  its  own 
performance  relative  to  a  care  standard;  and  those 
that  are  conducted  by  or  for  the  use  of  external 
review  organizations,  both  to  document  variation  in 
performance  relative  to  a  standard  and  to  identify 
poor  performers. 

Internal  QA  reviews  have  generally  consisted  of 
manual  audits  of  paper  charts  or,  in  some  cases, 


computerized  patient  records  or  ancillary  support 
systems  such  as  the  laboratory  and  the  pharmacy. 
Reviews  of  individual  clinicians  by  external  accred- 
iting agencies  have  usually  been  based  on  manual 
chart  audits,  while  external  reviews  of  institutions 
or  health  plans  have  increasingly  utilized  auto- 
mated administrative  data  bases,  such  as  the  Uni- 
form Hospital  Discharge  Data  Set  (UHDDS) 
and/or  automated  inpatient  claims/billing  data.  A 
number  of  large  insurance  companies  have  used 
data  from  their  automated  medical  claims  data 
bases  to  evaluate  physicians'  practice  patterns  and 
to  select  cost-effective  physicians  for  their  pre- 
ferred provider  networks.  Hospitals,  health  mainte- 
nance organizations  (HMOs),  and  other  employers 
of  physicians  have  recently  been  given  access  to  the 
National  Practitioner  Databank,  a  data  base  con- 
taining information  on  physicians'  malpractice 
claims  experience,  to  help  evaluate  the  clinical 
competence  of  prospective  employees. 

Numerous  studies  have  used  automated  admin- 
istrative data  bases  to  study  various  aspects  of  the 
quality  of  care  provided  in  inpatient  settings  in  the 
United  States.  Those  that  focus  on  access  to  care 
may  also  provide  some  insights  into  the  quality  of 
outpatient  care  received  prior  to  admission. 

Several  studies  have  examined  geographic  varia- 
tions in  hospital  admission  rates  for  selected  diag- 
noses or  surgical  procedures  using  UHDDS  or  inpa- 
tient claims  data  (Campion,  Naessens,  Leibson,  and 
others,  1992;  Chassin,  Brook,  Park,  and  others, 
1986;  McMahon,  Wolfe,  and  Tedeschi,  1989;  Perrin, 
Homer,  Berwick,  and  others,  1989;  Roos, 
Wennberg,  and  McPherson,  1988;  Spitzer  and 
Caper,  1989;  Wennberg,  Freeman,  and  Culp,  1987; 
Wennberg,  Roos,  Sola,  and  others,  1987).  Some 
have  used  UHDDS  data  to  examine  the  relation- 
ship between  socioeconomic  variables  and  either 
small-area  variation  in  discharge  rates  (McLaughlin, 
Normolle,  Wolfe,  and  others,  1989)  or  rates  of 
selected  procedures  among  patients  discharged  with 
the  same  diagnosis  (Wenneker  and  Epstein,  1989; 
Wenneker,  Epstein,  and  Weissman,  1990).  Still  oth- 
ers have  studied  late  hospitalizations  (Louis, 
Gonnella,  and  Zelezuik,  1988),  potentially  avoid- 
able admissions  (Spitzer  and  Caper,  1989),  and  early 
readmissions. 

Other  automated  information  systems  used  to 
assess  access  to  care  include  patient  registration 
data  and  birth  records,  which  are  used  to  study 
adequacy  of  prenatal  care  (Krieger,  Connell,  and 
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LoGerfo,  1992)  and  tumor  registries  for  analyzing 
the  stage  of  breast  cancer  diagnoses  by  hospital 
(Hand,  Sener,  Imperato,  and  others,  1991)  and  ac- 
cess to  breast  conserving  surgery  and  radiation  ther- 
apy (Lazovich,  White,  Thomas,  and  others,  1991). 
These  two  latter  studies  also  used  tumor  registry 
data  to  assess  adequacy  of  treatment  and  followup. 
Within  this  quality  parameter,  peer  review  organiza- 
tions (PROs)  use  discharge  abstract  data  to  study 
rates  of  occurrence  of  preventable  complications. 

In  the  United  States,  the  lack  of  a  common 
patient  identifier  has  prevented  the  linking  of  data 
from  most  administrative  data  bases,  limiting  our 
ability  to  evaluate  care  across  settings.  This  makes 
the  assessment  of  postdischarge  treatment  and 
followup,  as  well  as  outcomes,  across  diverse 
patient  populations  difficult  or  impossible  for 
external  reviewers. 

Most  externally  based  outcome  studies  in  the 
United  States  have  also  focused  on  hospital  dis- 
charge abstracts  or  inpatient  claims  data,  particu- 
larly on  hospital  mortality  rates.  The  first  such 
studies,  done  by  the  Health  Care  Financing  Admin- 
istration (HCFA),  reported  crude  death  rates  by 
hospital,  adjusted  only  for  age  and  sex.  More 
recently,  methodologies  have  been  developed  to 
adjust  these  rates  for  patient  risk  based  on  other 
claims-based  data  fields.  Studies  of  risk-adjusted 
hospital  mortality  rates  have  still  shown  significant 
variation  by  hospital  (Blumberg,  1988;  DuBois, 
Brook,  and  Rogers,  1987;  Green,  Passman,  and 
Wintfeld,  1991;  Jencks,  Daley,  Draper,  and  others, 
1988;  Smith,  Pine,  Bailey,  and  others,  1991). 

Several  outcome  studies  found  a  positive  rela- 
tionship between  surgical  volume  and  outcome 
(Flood,  Scott,  and  Ewy,  1984;  Luft  and  Hunt,  1986; 
Showstack,  Rosenfeld,  Garnick,  and  others,  1987) 
and  recommended  the  regionalization  of  certain 
procedures  as  a  method  to  improve  outcomes  for 
these  types  of  patients.  Shortell  identified  character- 
istics of  hospital  markets,  as  well  as  organization 
and  control  issues  that  may  influence  outcomes 
(Shortell  and  Hughes,  1988).  Chassin,  Park,  Lohr, 
and  others  (1989)  showed  that  the  measure  of  mor- 
tality used  (inpatient  deaths,  deaths  within  30  days 
of  discharge,  etc.)  can  affect  the  degree  of  variability 
among  hospitals. 

The  Medicare  data  files,  which  may  be  linked 
through  a  common  patient  identifier,  permit  the 
measurement  of  postdischarge  mortality  among  the 
elderly  enrolled  in  Medicare.  However,  studies  in 


other  populations  suffer  from  the  inability  to  track 
postdischarge  mortality  via  automated  data  bases 
due  to  the  lack  of  common  patient  identifiers.  Such 
analyses  require  direct  followup  with  patients, 
whose  identities  may  be  held  confidential  by  their 
caregivers.  Even  when  patient  data  can  be  collected 
and  linked,  the  costs  are  significantly  higher  than 
electronic  linkage  to  automated  death  records 
would  have  been. 

Numerous  studies  using  the  Canadian  health 
care  system's  administrative  data  bases  (Roos  and 
Brazauskas,  1990;  Roos,  Cageorge,  Austen,  and 
others,  1985;  Roos,  Flowerdew,  Wajda,  and  others, 
1986;  Roos,  Roos,  and  Sharp,  1987;  Roos  and  Sharp, 
1988)  have  demonstrated  the  usefulness  of  a  com- 
mon patient  identifier  for  linking  automated  data 
bases  to  identify  quality  problems  that  span  health 
care  delivery  sites. 

However,  two  innovative  studies  mentioned  pre- 
viously have  demonstrated  the  potential  for  algo- 
rithmically  matching  patient  files,  based  on  common 
data  fields,  to  assess  quality  of  care.  These  include: 
linking  patient  registration  data  and  discharge 
abstracts  with  birth  records  to  assess  prenatal  care 
process  and  outcomes  (Krieger  and  others,  1992); 
and  linking  claims  data  with  tumor  registry  data  to 
assess  the  appropriateness  of  treatments  for  breast 
cancer  (Lazovich  and  others,  1991).  In  addition,  an 
automated  acquired  immunodeficiency  syndrome 
(AIDS)  data  base  has  been  created  in  Maryland  by 
linking  discharge  abstracts,  inpatient  billing  data, 
death  records,  and  community-based  data  to  study 
process  and  outcomes  of  care  (Hidalgo,  1990). 

QA  activities  using  automated  information 
systems  also  have  been  undertaken  within  some 
health  care  organizations.  In  most  cases,  these  activ- 
ities have  been  led  by  academic  researchers  based  in 
teaching  hospitals,  academic  group  practices,  or 
HMOs.  Quality  indicators  selected  by  the  organiza- 
tion, rather  than  imposed  by  external  regulatory 
agencies,  have  been  monitored  using  either:  individ- 
ual (or  combinations  of)  internal  administrative 
data  systems,  such  as  automated  discharge  abstracts, 
billing  systems,  or  ancillary  support  systems  (for 
example,  laboratory,  radiology,  and  pharmacy  sys- 
tems); or  computerized  patient  records  (alone  or  in 
combination  with  internal  administrative  data  or 
ancillary  support  systems). 

Data  from  these  internal  systems  usually  can  be 
linked  using  a  common  patient  identifier  within  the 
institution.  However,  incompatible  data  systems  and 
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the  lack  of  consistent  standards  for  data  exchange 
sometimes  limit  intra-institutional  linkage. 

The  usefulness  of  automated  data  bases  within 
institutions  for  monitoring  clinical  quality  indicators 
has  been  reported  widely.  Some  of  these  internally- 
driven  studies  have  monitored  indicators  of  access 
to  care,  such  as:  analysis  of  the  appropriateness  of 
patient  admissions  to  scarce  ICU  beds  (Singer,  Carr, 
Mulley,  and  Thibault,  1983);  analysis  of  the  appro- 
priateness of  preoperative  selection  of  patients  with 
curable  renovascular  hypertension  (Cugini, 
Di  Palma,  Battisti,  and  others,  1991);  analysis  of  the 
percent  of  persons  with  AIDS  who  received  a  trial 
of  azidothymidine  (AZT)  or  home  care  services 
(Coltin,  Wurtman,  Kuvin,  and  others,  1990);  and 
analysis  of  compliance  with  well-child  screening  and 
immunization  standards  in  an  HMO  (Kind,  1989). 

Other  indicator  studies  have  focused  on  recogni- 
tion and  documentation  of  diagnoses  and/or 
adequacy  of  treatment  or  followup,  including  studies 
of  the  documentation  of  a  hypertension  diagnosis 
and  adequacy  of  treatment  in  patients  with  recur- 
rent, elevated  blood  pressure  readings  (Barnett, 
Winickoff,  Morgan,  and  others,  1983)  and  the  docu- 
mentation of  an  anemia  diagnosis  and  appropriate 
treatment  in  children  with  low  hemoglobin  or  hema- 
tocrit levels  using  a  computerized  medical  record 
system  (Coltin,  Neisuler,  and  Lurie,  1976). 

In  addition  to  using  automated  information 
systems  to  retroactively  monitor  aspects  of  patient 
care,  several  studies  have  reported  internal  use  of 
automated  interventions  to  improve  deficiencies 
identified  through  QA.  However,  as  with  QA  moni- 
toring, most  QA  interventions  have  focused  on 
clinicians,  regardless  of  where  in  the  process  the 
observed  defects  may  have  originated.  Clinicians 
are  both  suppliers  of  medical  care  services  to 
patients  and  customers  of  the  organizational  sup- 
port systems  that  enable  them  to  carry  out  their  part 
of  the  care  process.  Most  QA  interventions  focus 
only  on  the  supplier  role  of  clinicians  and  ignore 
their  needs  as  internal  customers  in  the  process. 
Examples  include:  an  HMO  that  used  computerized 
medical  records  to  generate  reminders  to  clinicians 
on  failures  to  comply  with  standards  for  diagnosis, 
treatment,  and  followup  of  hypertension  (Barnett 
and  others,  1983)  and  syphilis  (Winickoff,  Coltin, 
Fleishman,  and  others,  1985);  three  hospital  emer- 
gency rooms  that  employed  an  encounter  data  base 
to  monitor  adherence  to  treatment  regimens  and 
provide  feedback  when  deviations  occur  (Brand, 


Frazier,  Clancy,  and  Freeman,  1980);  and  a  psychi- 
atric clinic  that  used  an  expert  system  to  alert  clini- 
cians to  deviations  from  drug  treatment  protocols 
(Bronzino,  Morelli,  and  Goethe,  1989). 

In  addition  to  individualized  patient  reminders, 
automated  information  systems  also  have  been  used 
to  generate  summary  practice  profiles.  These  pro- 
files are  fed  back  to  clinicians  to  encourage  compli- 
ance with  established  standards  of  care  (Chodroff, 
1990;  Douglass  and  Batchelor,  1989;  Knight, 
O'Malley,  and  Fletcher,  1987;  McAlister,  Correy, 
Tong,  and  others,  1986;  Sickles,  1990;  Winickoff, 
Coltin,  Morgan,  and  others,  1984). 

Many  QA  interventions  merely  use  automated 
feedback  systems  to  request  the  retrospective  repair 
of  existing  defects  and  to  exhort  clinicians  to  do 
better.  This  is  typical  of  the  "quality  by  inspection" 
approach  and  often  results  in  inefficiency  or  creates 
frustration  among  clinicians,  who  may  not  have 
been  responsible  for  the  defects  and  have  no  means 
to  prevent  future  occurrences.  Some  of  these  inter- 
ventions merely  add  to  the  cost  of  poor  quality.  In 
contrast,  automated  QI  interventions  generally  take 
the  form  of  organizational  changes  to  the  process 
of  care  to  facilitate  prevention  of  defects. 

Uses  of  Automated  Information 
Systems  to  Support  QI 

QI  efforts  may  focus  on  the  quality  of  the  output 
or  the  outcome  and/or  efficiency  of  a  process.  Those 
that  focus  on  clinical  care  processes  require  detailed 
information  about: 

1.  Inputs  to  the  process,  which  may  be  patient  or 
population  factors  such  as  demographics,  risk 
factors,  co-morbidities,  and  insurance  status. 

2.  Steps  and  characteristics  of  the  process,  such  as 
technical  process  actions,  interpersonal  factors, 
resource  and  training  factors,  and  time  depen- 
dencies. 

3.  Outcomes  of  the  process,  such  as  symptom 
levels,  complications,  laboratory  test  results, 
functional  status,  health  behaviors,  mortality, 
and  satisfaction  with  care. 

Generally,  the  detailed  data  required  for  QI  are 
available  only  at  the  local  level,  which  is  one  reason 
why  institutional  QI  efforts  have  been  internal  to 
a  given  care  setting.  Many  of  the  data  elements 
required  for  QI  are  contained  within  automated 
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clinical  information  systems  in  hospitals  or  other 
organized  health  care  settings  (Glaser,  1991;  Hunter 
and  Vanderslice,  1991).  Such  settings  generally 
assign  a  unique  identifier  to  their  patients.  Comput- 
erized medical  record  systems  and  multiple  adminis- 
trative systems — such  as  automated  laboratory, 
pharmacy,  or  billing  systems — have  been  linked 
within  institutions  for  QI  analyses  and  to  support 
interventions  at  the  institutional  level. 

Another  reason  for  the  institutional  focus  is  that 
QI  efforts  incorporate  the  responsibility  for  correct- 
ing and  preventing  deficiencies.  While  many  studies 
have  used  interventions  to  improve  performance, 
they  are  not  necessarily  QI  studies.  QI  studies 
involve  the  design  of  interventions  to  prevent 
process  failures  that  result  in  inefficiency  or  poor 
outcomes.  Thus,  the  intervention  is  applied  to  the 
process,  not  merely  the  physician  as  supplier. 

Several  studies  using  computerized  medical 
records  have  employed  automated  reminders  to 
clinicians  at  the  time  of  the  patient  encounter  to 
proactively  improve  compliance  with  diagnosis  and 
treatment  guidelines.  Examples  of  the  use  of  auto- 
mated information  systems  to  alert  clinicians  to 
possible  diagnosis  or  treatment  failures  in  time  to 
implement  corrective  measures  include: 

•  Two  studies  done  in  a  single  hospital  setting,  one 
of  which  used  a  combination  of  laboratory  test 
results  and  computerized  medical  records  to 
alert  physicians  to  life-threatening  conditions 
based  on  abnormal  laboratory  values  (Tate, 
Gardner,  and  Weaver,  1990)  and  another  that 
alerted  them  to  hospital  acquired  infections  and 
the  appropriateness  of  their  treatment  (Evans, 
Larsen,  Burke,  and  others,  1986). 

•  An  academic  medical  practice  that  used  proto- 
cols or  decision  rules  on  a  partially  computerized 
medical  record  system  to  remind  physicians  of 
indicated  actions  (McDonald,  1976;  McDonald, 
Hui,  Smith,  and  others,  1984). 

•  An  HMO  that  used  computerized  medical 
records  to  generate  concurrent  reminders  to 
clinicians  at  routine  prenatal  visits  on  actions 
needed  to  comply  with  prenatal  care  guidelines 
(Morgan,  Studney,  Barnett,  and  Winickoff,  1978) 
and  to  generate  computer  alerts  at  all  scheduled 
visits  of  high-risk  patients  on  their  flu  immuniza- 
tion status  (Barton  and  Schoenbaum,  1990). 


Automated  information  systems  also  have  been 
used  as  prospective  aids  in  clinical  decisionmaking 
by  supplying  predictive  data  on  the  likelihood  of 
either  adverse  events  (Blaufass  and  Tinker,  1988; 
Shabot,  Leyerle,  and  LoBue,  1987)  or  successful 
interventions  (Cugini  and  others,  1991).  In  addition, 
a  number  of  studies  have  reported  the  use  of  office- 
based  microcomputer  systems  for  tracking  preven- 
tive screening  procedures  and  generating  concurrent 
reminders  to  clinicians  or  prospective  reminders  to 
patients  of  the  need  for  periodic  screening  tests 
(Fordham,  McPhee,  Bird,  and  others,  1990;  Frame, 
1990;  Harris,  O'Malley,  Fletcher,  and  others,  1990). 

Still  others  have  linked  computerized  patient 
records  to  existing  clinical  support  data  bases 
(Gottlieb,  Margolis,  and  Schoenbaum,  1990; 
Kuperman,  James,  Jacobsen,  and  Gardner,  1991) 
or  developed  their  own  personal  computer-based 
information  networks  (Edsall,  1991)  to  capture 
detailed  and  time-specific  data  on  all  steps  in  a 
care  process,  report  on  process  performance, 
support  the  analysis  of  root  causes,  and  implement 
computer-based  interventions. 

In  organizations  with  automated  patient  data 
systems,  the  ready  availability  of  computerized  data 
may  lead  managers  or  clinicians  to  focus  narrowly 
on  the  uses  of  information  technology  for  improving 
all  health  care  processes,  when  a  QI  analysis  of  root 
causes  might  suggest  other  approaches  for  some 
types  of  deficiencies.  Likewise,  the  same  outcome 
or  inefficiency  defect  observed  in  different  care  set- 
tings could  have  totally  different  causes.  Therefore, 
an  automated  solution  in  one  setting  may  have  little 
effect  in  another.  Local  investigation  and  focused 
remedial  action  are  necessary. 

Additional  data  needs.  Care  for  a  given  health 
problem  often  can  involve  multiple  care  settings. 
Therefore,  institutionally  based  studies  are  limited 
in  that  they  may  be  able  to  improve  some  of  the 
important  processes  within  an  episode  of  care  but 
only  reduce  some  of  the  poor  outcomes.  Likewise,  if 
the  root  causes  of  institutional  process  defects  are 
found  to  lie  in  broader  policy  issues,  such  as  the 
design  of  health  insurance  benefits  or  reimburse- 
ment mechanisms,  local  solutions  are  unlikely  to  be 
found.  Here,  interinstitutional  QI  studies  would  be 
needed  to  establish  root  causes.  The  limited  imple- 
mentation of  computerized  medical  record  systems 
currently  leaves  us  dependent  on  administrative 
data  bases  to  address  the  data  needs  of  QI  activities. 
Some  of  the  more  detailed  administrative  data  bases 
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within  institutions  (for  example,  laboratory,  radiol- 
ogy, and  pharmacy  systems)  may  help  to  compen- 
sate for  the  lack  of  computerized  medical  records  in 
supporting  QI  activities. 

The  ability  to  link  such  data  bases  across  care 
settings  to  establish  both  the  chronology  and  the 
characteristics  of  a  care  process  would  greatly  facil- 
itate QI  efforts.  These  data  bases  also  could  be 
augmented  with  patient-reported  data,  since 
detailed  information  on  both  the  inputs  and  out- 
comes of  clinical  care  processes  frequently  are 
reported  inadequately  in  medical  records  and 
administrative  data  bases. 

Patients  long  have  been  recognized  and  heavily 
used  as  a  valuable  source  of  information  both  on 
their  own  inputs  to  the  care  process — such  as 
demographics,  socioeconomic  characteristics,  atti- 
tudes, preferences,  lifestyle  and  other  behaviors, 
and  personal  and  family  medical  history — and  on 
various  types  of  health  outcomes  such  as  symptom 
levels,  functional  and  health  status,  quality  of  life, 
and  satisfaction  with  care.  More  recently,  studies 
have  found  that  patients  are  a  credible  source  of 
information  on  the  clinical  care  process  as  well 
(Davies  and  Ware,  1988).  A  number  of  validated 
instruments  now  exist  for  measuring  various 
aspects  of  patients'  health  status  (Nelson  and 
Berwick,  1989),  as  well  as  patient  satisfaction  with 
care  (Nelson,  Hays,  Larson,  and  others,  1989; 
Ware,  Snyder,  and  Wright,  1976;),  and  a  number  of 
electronic  data  bases  containing  these  measures  for 
different  reference  populations  now  exist.  None  of 
these,  however,  is  currently  in  the  public  domain. 
The  availability  of  such  data  bases  for  benchmark- 
ing purposes  would  benefit  not  only  QI  efforts,  but 
likely  would  encourage  convergence  on  a  limited 
number  of  instruments  and  lead  to  the  growth  of 
additional  reference  population  data  bases. 

While  automated  information  technologies  for 
collecting  data  directly  from  patients  have  existed 
for  quite  some  time,  they  have  seen  limited  use  in 
patient  care  settings  (Roizen,  Coalson,  Hayward, 
and  others,  1992;  Thomas,  Fairbank,  Pynsent,  and 
Baker,  1989).  The  diffusion  of  these  technologies 
into  clinical  practice  could  help  not  only  to  reduce 
the  costs  of  data  collection,  but  also  to  improve  our 
ability  to  incorporate  data  on  patient-identified 
needs  and  preferences  into  real-time  clinical  deci- 
sionmaking. These  patient-reported  data  bases 
could  be  linked  with  those  containing  primary 
data  from  computerized  medical  record  or  other 


administrative  data  bases  to  support  QI  efforts.  To 
realize  these  opportunities  we  must  solve  the  prob- 
lems of  inconsistent  data  standards,  lack  of  a  unique 
patient  identifier,  and  lack  of  acceptable  safeguards 
for  patient  privacy  and  data  security. 

To  add  to  the  challenge,  data  quality  concerns 
have  been  noted  in  most  studies  using  automated 
administrative  data  bases.  Specifically,  coding  sys- 
tems are  currently  inadequate  for  distinguishing 
tentative  from  confirmed  diagnoses  (Steinwachs  and 
others,  1989),  coding  practices  may  differ  across 
institutions,  data  definitions  may  vary  by  institution 
or  payer,  and  data  completeness  and  accuracy  are 
suspect  (Blumberg,  1987, 1988;  Campion  and  oth- 
ers, 1992;  Steinberg,  Whittle,  and  Anderson,  1990). 
All  of  these  problems  introduce  biases  in  interpret- 
ing the  results  of  such  studies. 

Progress  in  meeting  data  needs.  Progress  is 
being  made  in  many  of  these  areas.  The  tenth  revi- 
sion of  the  International  Classification  of  Diseases 
(ICD-10)  is  expected  to  address  the  problem  of  ten- 
tative vs.  confirmed  diagnoses;  payers  are  increas- 
ingly requiring  standard  data  definitions  and  coding 
practices;  and  reimbursement  mechanisms  that  are 
dependent  upon  complete  and  accurate  data  to 
assign  the  proper  payment  are  creating  incentives 
for  institutions  and  clinicians  to  improve  their  cod- 
ing practices. 

Some  researchers  have  suggested  coding 
improvements  that  would  build  upon  current  diag- 
nosis and  procedure  coding  schemes  at  marginal  cost 
to  greatly  increase  the  information  content  of  these 
codes,  thereby  aiding  QI  efforts  (Janssen,  Viste,  and 
Gluck,  1990;  Mendenhall,  1987, 1991;  Naessens, 
Brennan,  Boberg,  and  others,  1991 ).  Others  have 
created  clear  and  specific  definitions  for  selected 
quality  and  utilization  "screening"  measures,  which 
may  facilitate  benchmarking  of  performance  to  "best 
in  class"  institutions  or  health  plans  (Rutstein, 
Berenberg,  Chalmers,  and  others,  1976;  Rutstein, 
Mullan,  Frazier,  and  others,  1983;  Weiner,  Powe, 
Steinwachs,  and  others,  1990),  although  some  have 
found  these  measures  to  be  overly  sensitive  and 
insufficiently  specific  (Sanazaro  and  Mills,  1991). 

Those  with  access  to  linked  automated  data 
bases  within  institutions  have  developed  innovative 
methods  for  using  "downstream"  data  to  identify 
"upstream"  quality  problems  (Louis  and  others, 
1988;  Mennemeyer  and  Winkelman,  1991),  while 
Canadian  researchers  have  developed  methodol- 
ogies for  linking  inpatient  and  outpatient  claims  to 
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track  quality  problems  across  care  settings  (Roos 
and  others,  1985;  Roos  and  others,  1987).  These 
methodologies  will  be  useful  in  the  future,  should 
linkage  become  possible  in  the  United  States. 

Code  "mapping"  strategies  between  ICD-9-CM 
(International  Classification  of  Diseases,  Ninth 
Revision,  clinically  modified)  and  CPT-4  (Current 
Procedural  Terminology,  Fourth  Edition)  procedure 
codes,  where  possible,  would  be  highly  useful  for 
linking  inpatient  and  outpatient  claims  data.  Agree- 
ment between  the  physician  claim  and  the  inpatient 
or  surgical  day  care  claim  from  the  hospital  could 
help  validate  coding  accuracy  for  procedures  per- 
formed in  institutions. 

Information  Technologies  with  QA  and 
Ql  Applications 

Information  technology  may  be  defined  as  soft- 
ware, hardware,  and  information  science  theory 
applied  to  data  bases  and  information  systems  to 
expand  their  ability  to  meet  users'  needs. 

Information  technology  in  general  use  today 
improves  our  ability  to  assess  what  is  occurring  in 
health  care  delivery  but  does  not  substantially 
improve  our  understanding  of  why  certain  health 
care  processes  and  outcomes  occur.  There  is  a 
broad  range  of  emerging  technologies  in  prototype 
or  limited  production  that  can  be  incorporated  into 
QA  and  QI  systems.  These  innovations,  which  are 
being  created  by  medical  informatics  professionals 
and  adapted  from  other  industries,  will  help  answer 
both  the  "what"  and  the  "why"  of  health  care.  At 
the  stage  of  conceptual  discussion  are  additional 
products  of  information  science  and  theory,  poten- 
tial technologies  whose  role  in  QA  and  QI  is  yet 
to  be  clearly  defined. 

Information  technologies  currently  incorporated 
into  QA  and  QI  systems.  The  most  significant 
impact  of  current  information  science,  technology, 
and  statistical  analysis  is  their  effect  on  our  ability  to 
exploit  coded  data  contained  in  the  electronic  data 
bases  discussed  above.  Four  specific  aspects  of  this 
are  codification  of  data,  case  mix  systems,  automated 
medical  records,  and  expert  systems.  Codification  of 
data  has  impact  on  both  administrative  and  clinical 
systems  and  is  augmented  by  use  of  controlled 
vocabulary  and  structured  text  strings.  Case-mix 
systems  allow  adjustment  for  severity  of  illness  and 
grouping  according  to  resource  utilization.  Auto- 
mated medical  records  have  been  implemented  at 


a  limited  number  of  sites,  but  they  have  become 
fundamental  to  the  operation  of  health  care 
delivery  at  those  sites.  Expert  systems  applications 
fall  in  two  categories:  monitors  that  offer  retrospec 
tive  reminders  and  consultants  who  offer  off-line 
decision  support. 

Codification  of  data.  Codification  of  clinical 
data  facilitates  comparison  of  data  when  collected 
by  multiple  providers  or  over  time.  When  data 
systems  are  automated,  codification  is  necessary  to 
take  advantage  of  the  rapid  processing  speed  and 
integration  capabilities  that  computers  provide. 
Several  authors  have  commented  on  the  advantage 
of  coding  in  health  care,  particularly  in  light  of  the 
lack  of  a  standardized  nomenclature  (Bishop  and 
Dombrowski,  1990;  McDonald,  1991;  Shortliffe, 
Perreault,  Widerhold,  and  Fagan,  1990). 

In  the  United  States,  the  most  widely  used  med 
ical  coding  schemes  include  the  following: 

•  ICD-9-CM  supported  by  the  World  Health 
Organization  for  inpatient  discharge  diagnoses. 

•  CPT-4  supported  by  the  American  Medical 
Association  for  physician  services  procedures. 

•  DSM-III  (Diagnostic  and  Statistics  Manual, 
Third  Edition)  supported  by  the  American 
Psychiatric  Association  for  classification  of 
mental  disorders  incorporating  measures  of 
stressors  and  function. 

•  HCPCS  (HCFA  Common  Procedure  Coding 
System)  for  procedures  for  Medicare  claims 
processing. 

•  SNOMED  (Systematized  Nomenclature  for 
Medicine)  supported  by  the  American  College 
of  Pathologists  for  pathologic  diagnoses  and 
workload  codes. 

•  AHFS  (American  Hospital  Formulary  Service) 
for  categorization  of  medications  and  prescrip- 
tion items. 

•  NDC  (National  Drug  Code)  supported  by  the 
Food  and  Drug  Administration  for  manufac- 
turer and  preparation-specific  medication 
identification. 

•  ICCS  (International  Classification  of  Clinical 
Services)  supported  by  Healthcare  Knowledge 
Resources  for  health  technology  services. 
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These  coding  schemes  are  fundamental  to  the 
work  of  QA  professionals.  However,  there  is  little 
in  the  literature  that  discusses  the  specific  effects  of 
coding  systems  on  QA.  One  exception  has  been  the 
use  of  ICCS  billing  data  to  monitor  and  improve 
hospital  drug  use  (Mendenhall,  1987, 1991). 

Each  of  the  systems  listed  above  uses  numeric 
or  alpha-numeric  strings  as  arbitrary  symbols  to 
represent  the  words  that  identify  an  object, 
process,  or  concept.  This  is  not  the  only  method  of 
codification  of  medical  information,  however,  as 
controlled  vocabulary  and  structured  text  also 
serve  this  purpose. 

In  a  controlled  vocabulary  (CV),  strings  of 
characters  that  appear  to  be  regular  words  or 
abbreviations  are  used  to  represent  what  the  words 
themselves,  or  other  words,  represent.  Similar  to 
alpha-numeric  codes,  these  coding  systems  do  not 
permit  deviation  in  content  (spelling)  or  syntax 
when  a  specific  meaning  is  desired. 

In  some  situations,  the  CV  serves  as  a  standard- 
ized dictionary  of  usable  terms,  each  uniquely 
linked  to  one  machine  usable  code  (Clark  and  Shea, 
1990;  Masarie,  Miller,  Bouhaddou,  and  others, 
1991).  For  example,  "thoracentesis"  is  uniquely 
linked  to  the  ICD-9-CM  code  34.91.  Automated 
systems  may  accept  and  report  the  procedure 
thoracentesis  using  either  CV  or  code,  although 
it  is  unlikely  the  CV  itself  would  be  stored  and 
processed. 

In  other  situations,  the  CV  terms  themselves  are 
stored  and  processed  in  the  automated  information 
system.  Frequently  used  CVs  are  units  of  measure, 
such  as  "mg"  and  "cc,"  as  well  as  positional  modi- 
fiers such  as  "left"  and  "sit." 

Structured  text  is  a  technique  by  which  meaning 
is  assigned  to  and  extracted  from  letters  and  num- 
bers according  to  their  position  in  a  character  string. 
If  each  segment  is  always  of  fixed  length,  it  can  be 
stored  and  processed  according  to  its  character  posi- 
tion in  the  string,  like  the  columns  in  old-fashioned 
punched  cards.  If  the  length  of  individual  segments 
can  vary,  delimiters  are  necessary  for  the  informa- 
tion system  to  distinguish  the  segments. 

Case-mix  systems.  Case-mix  systems  are 
designed  to  categorize  patients  according  to  signifi- 
cant characteristics  of  their  medical  condition  or 
care  received.  Using  advanced  analytic  techniques 
in  their  creation,  and  manipulation  of  enormous 
administrative  and  clinical  data  bases  in  their  design 
and  validation,  case-mix  systems  are  a  second  area 


in  which  information  technology  has  an  impact 
on  QA.  Case-mix  systems  generally  fall  into  two 
varieties:  those  that  classify  according  to  resource 
consumption  and  those  that  classify  according  to 
severity  of  illness. 

The  Medicare  prospective  payment  system  is 
anchored  on  the  diagnosis-related  group  (DRG) 
case-mix  system.  DRGs  classify  patients  of  similar 
clinical  type  according  to  a  small  number  of  patient 
attributes  and  treatment  processes  found  in  the 
UHDDS.  Inpatient  resource  consumption  is  pre- 
dicted by  DRG,  which  then  serves  as  a  means  to 
define  hospital  case-mix  for  internal  analysis  and 
comparison  between  facilities.  DRGs  thus  serve  as 
a  common  denominator  for  inpatient  QA  analyses. 
Case-mix  systems  for  analysis  of  ambulatory 
resource  consumption  are  under  active  develop- 
ment both  within  the  United  States  and  abroad 
(Greenlick,  1991;  Weiner,  1991;  Wood,  1991). 

Measurement  of  inpatient  quality  of  care  is 
improved  when  patient  severity  of  illness  is  consid- 
ered more  fully  than  possible  with  DRGs.  It  then 
becomes  more  feasible  to  answer  the  question: 
"Who  is  caring  for  the  sickest  patients?"  Severity 
systems  allow  deeper  analysis  of  the  effectiveness, 
appropriateness,  and  efficiency  of  care  rendered  by 
considering  the  effects  of  comorbidity  and  compli- 
cations, as  well  as  permitting,  in  some  systems, 
review  of  patient  status  concurrently  and  repeatedly 
during  their  admission. 

A  number  of  commercial  products  provide  tools 
for  severity  assessment.  Some  are  fully  automatable 
because  they  use  only  electronically  available 
UHDDS  elements  to  determine  severity.  Others 
incorporate  detailed  clinical  findings  into  their 
severity  analysis,  requiring  manual  chart  review. 
Several  system  vendors  also  provide  national  com- 
parative data  bases  for  license  holders.  Commercial 
severity  of  illness  measurement  systems  have  been 
cataloged  or  compared  in  the  literature  (Aronow, 
1988;  Fifer,  1987;  Horn  and  Buckle,  1989).  Noncom- 
mercial case-mix  systems  and  their  value  in  QA 
also  have  been  described  (Janssen  and  others,  1990; 
Shabot  and  others,  1987). 

As  noted  by  Fifer  (1987),  the  analytic  powers  of 
computers  have  permitted  the  emergence  of  case- 
mix  systems  and  allowed  QA  to  enter  "the  era  of 
norms."  However,  continued  improvement  is 
needed  in  the  design  and  use  of  case-mix  systems 
to  account  for  additional  attributes  of  patient 
diversity  (Green  and  others,  1991)  and  increase  the 
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range  of  electronic  clinical  data  to  be  integrated 
into  case-mix  systems  (Shabot,  Bjerke,  LoBue, 
and  others,  1991). 

Automated  medical  records.  The  third  area  in 
which  information  technology  has  affected  QA  and 
QI  is  the  automated  medical  record  (AMR)  (Dick 
and  Steen,  1991;  General  Accounting  Office,  1991; 
Shortliffe  and  others,  1990).  Currently,  AMRs  have 
been  implemented  in  a  limited  number  of  clinical 
settings.  With  the  exception  of  those  few  centers 
where  AMRs  have  been  embraced  enthusiastically 
for  a  number  of  years,  the  AMR  should  be  viewed 
as  an  emerging  technology.  However,  because  at 
these  institutions  the  AMR  is  not  an  academic  pro- 
totype, but  is  relied  upon  to  meet  everyday  opera- 
tional needs,  they  are  listed  below.  General  descrip- 
tions and  bibliographies  of  these  systems  are 
available  in  recent  publications  (Agency  for  Health 
Care  Policy  and  Research,  in  press;  Dick  and  Steen, 
1991;  Shortliffe  and  others,  1990;  Stead,  1989). 

•  AMRS  (Automated  Medical  Record  System) 
adapted  from  COSTAR IV  and  implemented  at 
the  Harvard  Community  Health  Plan. 

•  Beth  Israel  and  Brigham  and  Women's 
Hospitals'  system. 

•  COSTAR  (Computer-Stored  Ambulatory 
Record)  developed  at  Massachusetts  General 
Hospital  and  implemented  at  several  hundred 
sites  worldwide,  currently  in  its  fifth  release. 

•  DHCP  (Decentralized  Hospital  Computer  Pro- 
gram) developed  and  implemented  at  nearly  200 
sites  by  the  Department  of  Veterans  Affairs. 

•  HELP  (Health  Evaluation  Through  Logical 
Processing)  developed  and  implemented  at 
Latter  Day  Saints  Hospital. 

•  RMRS  (Regenstrief  Medical  Record  System) 
developed  and  implemented  at  Indiana  Univer- 
sity Medical  Center. 

•  STOR  (Summary  Time  Oriented  Record) 
developed  and  implemented  at  the  University 
of  California,  San  Francisco. 

•  THERESA  developed  privately  for  and  imple- 
mented at  Grady  Memorial  Hospital. 

•  TMR  (The  Medical  Record)  developed  and 
implemented  at  Duke  University  Medical 
Center. 


Compared  with  paper  medical  records,  several 
characteristics  of  AMRs  allow  them  to  enhance 
QA  and  QI:  greater  accessibility,  more  rapid  data 
retrieval,  more  complete  and  consistent  organiza- 
tion, greater  readability,  better  integration  of 
results  reports,  and  the  potential  to  be  more  com- 
plete and  accurate. 

The  most  important  benefits,  however,  are  the 
ad  hoc  query  and  monitoring  functions  AMRs 
allow,  functions  that  have  no  corollary  in  the  paper 
record.  As  a  query  tool  in  QA,  the  AMR  allows  the 
clinical  records  of  large  patient  populations  to  be 
searched  according  to  specified  criteria  to  produce 
reports  at  various  degrees  of  summarization. 
Queries  often  provide  the  means  for  defining  study 
groups  for  QA  projects.  Special  high-level  query 
languages  have  been  developed  for  AMRs,  includ- 
ing Clinquery  at  the  Beth  Israel  Hospital,  the  Med- 
ical Query  Language  (MQL)  for  COSTAR, 
STRATO  for  HELP,  and  CARE  and  FAST  for  the 
RMRS  (Adams,  1986;  McDonald,  Tierney,  Martin, 
and  others,  1991;  Pryor,  1988;  Safran,  Porter,  Rury, 
and  others,  1990).  Several  recent  reports  describe 
these  activities  (Hoffman,  Christenson,  Brimley,  and 
others,  1990;  Payne,  Goodson,  Morgan,  and  others, 
1989).  AMR  monitoring  functions  are  discussed  in 
the  next  section. 

Expert  systems.  Expert  systems  are  a  fourth  area 
in  which  information  technology  is  now  incorpo- 
rated into  QA  and  QI.  As  defined  by  Shortliffe  and 
colleagues  (1990,  p.  653),  expert  systems  symboli- 
cally encode  concepts  derived  from  experts  in  a 
field  and  use  that  knowledge  "to  provide  the  kind  of 
problem  analysis  and  advice  that  the  expert  might 
provide."  Thus,  the  expert  system  approach  is  con- 
ceptually very  general  and  is  applicable  at  many  lev- 
els of  complexity. 

Two  types  of  expert  systems  may  be  distin- 
guished: monitors  providing  retrospective  reminders 
and  consultants  offering  off-line  decision  support. 
There  may  be  overlap  of  these  functions  in  specific 
applications,  and  the  current  range  of  application  of 
expert  systems  will  expand  with  the  emerging  tech- 
nologies discussed  later  in  this  paper. 

Monitors  provide  automatic  surveillance  of  the 
clinical  data  of  active  patients  to  look  for  individuals 
who  have  specified  data  occurrences  or  patterns  of 
data  occurrences.  These  have  been  found  particu- 
larly useful  in  the  delivery  of  preventive  health 
maintenance.  Positive  findings  (defined  by  provider- 
authored  rules)  generate  alerts,  reminders,  or  other 
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kinds  of  feedback  to  providers.  Expert  monitors  are 
not  restricted  to  AMRs,  but  their  most  significant 
applications  are  intimately  linked  to  AMRs.  Many 
studies  of  expert  system  surveillance  contributing 
to  QA  have  been  reported,  particularly  for  three 
pioneer  AMRs:  COSTAR,  HELP,  and  RMRS 
(Barnett,  Winickoff,  Dorsey,  and  others,  1978; 
Barnett  and  others,  1983;  Berwick  and  Coltin, 
1986;  Bradshaw,  Gardner,  and  Pryor,  1989;  Evans, 
Gardner,  Bush,  and  others,  1985;  Evans,  and  others, 
1986;  Evans,  Burke,  Pestotnik,  and  others,  1990; 
Frame,  1990;  Harris  and  others,  1990;  McDonald, 
1976;  McDonald  and  others,  1984;  Norman  and 
Hardin,  1990;  Ornstein,  Garr,  Jenkins,  and  others, 
1991;  Pestotnik,  Evans,  Burke,  and  others,  1990; 
Tate  and  others,  1990;  Winickoff  and  others,  1984). 

Consultant  expert  systems  offer  the  clinical  com- 
petence of  their  expert-author  panels  on  an  as- 
needed  basis  to  health  care  providers.  Although 
there  are  many  such  systems  in  use,  few  have  been 
reported  for  their  contribution  to  QA  and  QI  activi- 
ties (de  Dombal,  1989;  Evans  and  others,  1990). 
Currently,  providers  use  the  systems  for  assistance 
in  caring  for  individual  patients  rather  than  the 
process  and  population-based  needs  represented  in 
QA  and  QI  (Carlson,  1989). 

Essential  to  the  operation  of  any  type  of  expert 
system  is  the  knowledge  base  (KB)  wherein  the  sys- 
tem's expertise  resides.  Included  here  are  the  logic 
rules,  weights,  facts,  and  models  through  which 
expertise  is  computerized.  KBs  are  discussed  in  sev- 
eral recent  publications  (Giuse,  Giuse,  and  Miller, 
1990;  Holbrook,  Langton,  Haynes,  and  others,  1991; 
Lincoln,  1990;  Perry,  1990) 

Emerging  information  technologies.  Emerging 
information  technologies  have  tremendous  poten- 
tial to  enhance  QA  and  QI  efforts  in  health  care, 
particularly  for  connectivity  and  use  of  encounter 
data.  Connectivity  is  the  integration  into  compre- 
hensive health  care  information  systems  of  data, 
information,  and  knowledge  across  systems,  sites, 
and  time.  Issues  include: 

•  Local  and  regional  networking. 

•  Centralized  vs.  distributed  systems. 

•  Proliferation  of  AMRs. 

•  Data  accessibility  and  portability. 

•  Workstations  as  resource  centers  and  integration 
tools. 


•  Matching  of  patient  identifiers. 

•  Development  of  standards  for  use  of  medical 
data. 

Increased  connectivity  will  substantially  improve 
the  ability  of  QA  to  identify  what  is  occurring  in 
health  care.  A  number  of  discussions  of  connectivity 
issues  and  benefits  to  QA  and  QI  have  been 
published  (Ball  and  Douglas,  1991;  Cimino  and 
Sengupta,  1991;  Dick  and  Steen,  1991;  Glaser,  1991; 
Kamel  and  Zviran,  1991;  Lansky,  1989;  Roos  and 
Sharp,  1988;  Selbmann  and  Pietsch-Breitfeld,  1990; 
Shortliffe,  1991;  Stead,  1991;  Zinn,  1990).  Addition- 
ally, many  implementations  of  integrated  health 
care  information  systems  have  been  reported 
(Albright,  Panzer,  Tuttle,  and  others,  1990;  Bakker, 
1990;  Branger  and  Duisterhout,  1991;  Edsall,  1991; 
Hidalgo,  1990;  Lansky,  1989;  Leyerle,  LoBue,  and 
Shabot,  1990;  McDonald  and  others,  1991;  Peng, 
VanDorsten,  Dilzer,  and  others,  1991;  Price, 
Chandrasekhar,  and  Tamirisa,  1990;  Pryor,  1988; 
Roos  and  Sharp,  1988;  Safran,  Slack,  and  Bleich, 
1989;  Scherrer,  Baud,  Hochstrasser,  and  Ratib,  1990: 
Weissman,  Mossel,  Hairnet,  and  others,  1990). 

Use  of  encounter  data  involves  the  collection, 
storage,  and  interpretation  of  data  from  the  private 
meeting  of  provider  and  patient.  Issues  include: 

•  Incorporation  of  encounter  or  progress  notes 
into  the  automated  medical  record. 

•  Data  entry  by  providers. 

•  Data  entry  by  patients. 

•  Processing  of  text  data. 

•  Structure  of  the  AMR. 

Encounter  data  include  data  that  the  provider 
records  as  history  and  physical  examination,  the 
logic  the  provider  followed  to  reach  documented 
assessments  and  diagnoses,  and  the  rationale  for 
orders  for  diagnostic  and  therapeutic  actions.  These 
data  will  provide  a  substantial  boost  to  QI  efforts  to 
determine  the  "why"  of  health  care.  Recent  publica- 
tions have  discussed  these  issues,  in  particular  the 
problems  surrounding  automation  of  provider- 
collected  data  and  the  effect  of  clinical  encounter 
data  on  QA  and  QI  (Dambro  and  Weiss,  1988;  Dick 
and  Steen,  1991;  General  Accounting  Office,  1991; 
Hammond  and  Stead,  1991;  Safran,  Rusy,  Rind,  and 
Taylor,  1991).  Issues  involved  with  complementary 
data  entry — data  from  the  patient's  encounter 
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experience — also  have  been  discussed  (Thomas  and 
others,  1989;  Williams,  Imrey,  and  Williams,  1991). 

Interest  in  both  encounter  data  and  quality  of 
care  have  contributed  to  experimentation  with  a 
variety  of  AMR  formats  and  data  collection  mecha- 
nisms (Bishop,  1991;  Hammond,  Straube,  and 
Stead,  1990;  Rodewald,  Szilagyi,  Wrenn,  and  Slovis, 
1991),  as  well  as  the  growing  interest  in  AMRs 
evidenced  by  the  recent  Institute  of  Medicine  report 
on  computer-based  patient  records  (CPRs)  (Dick 
and  Steen,  1991). 

Specific  emerging  information  technologies  may 
impact  on  connectivity  or  data  use  or  both.  Their 
potential  roles  will  be  described  in  terms  of  their 
place  in  the  following  QA  and  QI  process.  Technolo- 
gies that  pertain  to  more  than  one  step  in  the  process 
are  discussed  in  the  step  where  they  first  appear. 

Data  capture       Information  Knowledge 
codification-*      processing-*  dissemination 
and  storage 

Data  capture,  codification,  and  storage.  The 
capture  and  codification  of  data  for  administrative 
and  clinical  health  care  information  systems  provide 
inputs  for  QA  and  QI  processes.  Information  tech- 
nologies for  these  activities  may  be  grouped  as 
input  mechanisms,  standards  conventions,  and 
analytic  systems. 

Emerging  input  mechanisms  allow  more  rapid, 
complete,  flexible,  and  quality-checked  data  capture 
than  is  achievable  by  a  trained  intermediary  using 
paper  and  pen  in  a  hardcopy  record  or  AMR  data 
entry.  They  include  scanning,  pointing,  speech 
recognition,  handwriting  recognition,  image  digitiz- 
ing, bedside  terminals,  and  provider  workstations. 

Scanning,  using  an  optical  mark  reader,  identi- 
fies "bubbles"  filled  in  on  a  form  and  interprets 
them  according  to  their  location.  This  allows  rapid, 
error-checked  data  entry  to  mainframe  or  micro- 
computer information  systems  (Lindemann,  Rock, 
Locke,  and  Johnson,  1991).  Optical  character  read- 
ers scan  printed  text  and  translate  the  characters 
into  American  Standard  Code  for  Information 
Interchange  (ASCII)  code  for  storage,  retrieval, 
and  possible  analysis.  Bar  code  readers  translate 
printed  lines  to  positively  identify  people  and 
objects,  allowing  rapid  documentation  of  specimens, 
medications,  or  other  materials  handled,  with  anno- 
tations of  persons  and  places. 

Pointing  devices  include  the  mouse,  touch 
screens,  and  light  pens  which  may  be  incorporated 


into  the  interface  of  workstations  or  portable  termi- 
nals. Pointers  allow  use  of  icons  and  menu  displays 
to  speed  communication  with  computer  systems 
(McDonald,  1990). 

Automated  speech  recognition  (ASR)  is  con- 
cerned with  the  comprehension  of  spoken  input  and 
the  translation  of  that  input  into  the  electronic 
medium  (Belton  and  Dick,  1991;  Bergeron  and 
Locke,  1990;  McDonald,  1990).  ASR  should  not  be 
confused  with  voice  recognition,  which  is  concerned 
with  proper  identification  of  a  particular  speaker. 
Commercial  ASR  systems  are  being  used  in  emer- 
gency medicine  and  radiology,  and  noncommercial 
systems  are  being  developed  in  other  specialties 
(Cass,  1991;  Holbrook,  and  Ontai,  1991;  Landau, 
Norwich,  and  Evans,  1989;  Smith,  Brien,  Pettus,  and 
others,  1990).  Although  obstacles  remain,  this  tech- 
nology holds  promise  for  providing  the  first  means 
of  direct  provider  entry  into  the  AMR  without  sig- 
nificant alteration  of  work  flow. 

Handwriting  recognition,  sometimes  referred  to 
as  pen  and  pad  technology,  is  combined  with 
portable  terminals  and  pointing  devices.  Input  is 
made  by  hand  printing  characters  on  forms  visible 
on  the  computer  screen.  This  technology  is  mini- 
mally developed,  but  it  holds  the  highest  potential 
for  allowing  providers  to  enter  data  into  the  AMR 
with  minimal  disruption  to  their  work  style 
(Andreshak,  Lumelsky,  Chang,  and  others,  1990; 
Bergeron  and  Locke,  1990;  McDonald,  1990). 

Image  digitizing  and  processing  has  been  devel- 
oping rapidly  due  to  the  highly  computerized  nature 
of  many  medical  imaging  procedures.  Picture 
archiving  and  communications  systems  (PACS)  are 
becoming  more  common  in  radiology  departments 
and  can  be  expected  to  spread  to  other  image-rich 
specialties,  such  as  ophthalmology  and  pathology. 
These  systems  allow  automated  storage,  retrieval, 
and  manipulation  of  images,  making  them  available 
to  multiple  providers  at  desktop  workstations  in  dis- 
tant locations.  Image  analysis  and  reporting  systems 
are  now  being  prototyped  (Dayhoff,  Maloney, 
Kenny,  and  Fletcher,  1991;  McDonald,  1989b;  Ratib 
and  Huang,  1991;  Rowberg  and  Price,  1991; 
Rusinek  and  Mourino,  1989). 

Bedside  terminals — both  hand-held  portable 
devices  and  those  dedicated  to  specific  patient 
beds — are  designed  to  allow  data  capture  at  the 
time  and  place  the  data  are  generated  (Ball  and 
Douglas,  1991;  Higgins,  Jiang,  Swindell,  and 
Bernard,  1991;  Nowlan,  Rector,  Kay,  and  others, 


63 


1990).  Among  other  goals,  these  "point  of  care  sys- 
tems" are  expected  to  minimize  recordkeeping  time, 
reduce  errors  due  to  manual  transcription,  perform 
calculations,  and  check  data  for  errors.  They  gener- 
ally feature  graphic  user  interfaces,  and  many  com- 
mercial models  are  available. 

Provider  workstations  incorporate  many  of  the 
technologies  discussed  above  into  an  intelligent  ter- 
minal or  stand-alone  computer  with  high-resolution 
graphic  interface.  The  workstation  itself  may  be  the 
means  to  achieve  integration  of  heterogeneous 
information  systems,  especially  for  information 
retrieval  by  practitioners  (Annevelink,  Young,  and 
Tang,  1991;  Hammond,  Berger,  Carey,  and  others, 
1991;  McCallie,  Margulies,  Kohane,  and  others, 
1990;  Silva,  Zawilski,  O'Brian,  and  others,  1990). 
Like  bedside  terminals,  workstations  may  create  an 
environment  conducive  to  provider  data  entry  (Fan, 
Machlin,  Wang,  and  Chang,  1990).  Electronic  mail 
allows  communication  with  other  providers,  alerts, 
reminders,  and  reports.  Bibliographic  resources  are 
available,  as  well  as  other  reference  materials. 

Standards  conventions  are  in  development  for 
coded  medical  data  and  for  definition  of  data  sets, 
data  exchange  practices,  and  the  semantics  of 
knowledge  representation.  These  efforts  are  mutu- 
ally dependent  and  affect  information  processing 
and  knowledge  dissemination  for  QA  and  QI,  as 
well  as  data  capture  (Agency  for  Health  Care  Policy 
and  Research,  1991;  McDonald,  1991). 

Many  standards  exist  for  the  codification  of 
medical  data,  as  noted  previously.  However,  these 
are  inconsistent  in  their  level  of  detail  and  the  com- 
pleteness with  which  they  cover  their  specialty.  Fur- 
thermore, there  are  incompatibilities  between  code 
systems  for  similar  domains,  such  as  procedure 
codes  in  ICD-9-CM  and  CPT-4.  Finally,  codes  are 
lacking  for  data  obtained  in  history  taking  and  phys- 
ical examination. 

Some  of  these  problems  are  being  addressed  by 
the  organizations  discussed  in  the  data  exchange 
portion  of  this  section,  as  well  as  the  American  Soci- 
ety for  Testing  and  Materials  (ASTM),  which  is 
developing  a  chemistry  test  nomenclature.  Other 
deficiencies  will  be  improved  with  the  development 
of  new  minimum  health  care  data  sets. 

Currently,  health  care  data  sets  exist  formally  as 
the  UHDDS  and  informally  as  the  HCFA  claim 
forms:  UB-82  for  inpatient  services  and  HCFA-1500 
for  outpatient  and  physician  services.  However,  the 
UHDDS  consists  of  a  limited  demographic  patient 


profile  with  diagnostic  and  procedure  codes,  and 
completion  of  many  of  the  claims  form  fields  is 
optional. 

The  development  of  several  data  sets  is  noted  in 
the  Agency  for  Health  Care  Policy  and  Research 
(AHCPR)  (1991)  report  on  linking  research-related 
data  bases: 

•  The  Uniform  Ambulatory  Care  Data  Set 
(UACDS)  seeks  to  standardize  data  on  out- 
patient care  from  physicians  and  other  providers. 

•  The  Long-Term  Care  Facilities  Minimum  Data 
Set  seeks  to  describe  settings  of  long-term  care. 

•  The  Long-Term  Care  Client  Uniform  Data  Set 
seeks  to  describe  clients  of  long-term  care. 

•  The  Nursing  Facility  Resident  Assessment  Core 
Data  Set  seeks  to  promote  uniform  clinical  and 
functional  assessment  of  nursing  home  residents. 

Other  data  sets  in  development  include: 

•  The  Uniform  Clinical  Data  Set  (UCDS)  seeks 
to  provide  patient  risk  factor,  procedure,  and 
outcomes  data  for  expert  system  analysis  by 
PROs  to  improve  their  capabilities  for  monitor- 
ing health  care  quality  (Harrop,  1991;  Health 
Care  Financing  Administration,  1991). 

•  The  Wisconsin  Ambulatory  Review  Program 
(WARP)  seeks  to  develop  a  matrix  of  clinical 
findings,  attributes  applicable  to  the  findings, 
and  intensity  scales  for  the  attributes  for  analyz- 
ing ambulatory  care  quality  (Yanke  and 
Moskowitz,  1992). 

•  The  Nursing  Minimum  Data  Set  (NMDS)  seeks 
to  define  a  core  set  of  data  for  assessing  nursing 
practice  across  settings  (Delaney  and  Mehmert, 
1990;  Werley,  Devine,  Zorn,  and  others,  1991; 
Williams,  1991). 

The  only  data  exchange  or  message  standard 
currently  in  prominent  use  is  the  ASCII  code  which 
operates  at  the  individual  character  level.  Several 
organizations  are  actively  designing  medical  data 
exchange  standards  at  higher  levels  to  facilitate 
communication  within  and  between  health  care 
facilities  (Agency  for  Health  Care  Policy  and 
Research,  1991;  McDonald,  1991;  McDonald  and 
others,  1991;  McLinden,  D'Ascenzo  Carlos,  and 
Oleson,  1990;  Shortliffe,  1991).  These  include: 
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•  ACR/NEMA  (American  College  of  Radiol- 
ogy/National Electrical  Manufacturers  Associa- 
tion) is  developing  standards  for  PACS. 

•  ASTM  subcommittees  are  developing  standards 
for  physiologic  data,  reminder  systems  logic,  and 
data  exchange  between  computer  systems. 

•  CEN  (European  Standardization  Committee)  is 
developing  data  exchange  standards  in  Europe. 

•  HL7  (Health  Level  Seven)  health  care 
providers,  vendors  and  consultants  are  develop- 
ing exchange  standards  for  health  care  computer 
applications. 

•  MEDIX  (Medical  Data  Interchange  Committee 
of  the  Institute  of  Electrical  and  Electronic 
Engineers,  IEEE)  is  developing  exchange  stand- 
ards according  to  American  National  Standards 
Institute  (ANSI)  consensus  rules  (Harrington, 
Benson,  and  Spector,  1990). 

•  MIB  (Medical  Information  Bus  of  the  IEEE)  is 
developing  exchange  standards  between  medical 
devices  and  computers  (Nolan  and  Shabot,  1990). 

In  the  United  States,  many  vendors  and 
providers  have  announced  adoption  or  support  of 
interfaces  to  the  HL7  and/or  ASTM  standards. 
These  two  organizations  have  worked  closely  to 
make  their  efforts  complementary.  Communication 
between  medical  data  exchange  standards-setting 
organizations  has  been  provided  by  the  Healthcare 
Information  Standards  Coordinating  Committee, 
and  beginning  in  1992,  ANSI  also  will  help  coordi- 
nate standards-setting  efforts. 

Another  aspect  of  message  exchange  concerns 
standardization  of  network  systems.  The  Integrated 
Systems  Digital  Network  standard  attempts  to 
address  this  need  (Agency  for  Health  Care  Policy 
and  Research,  1991). 

A  final  level  of  standards  development  concerns 
exchange  of  medical  knowledge.  The  Arden  Syntax 
for  Medical  Logic  Modules,  derived  from  analysis  of 
the  knowledge  bases  of  the  HELP  and  Regenstrief 
ARMs,  has  been  proposed  (Hripcsak,  1991; 
Hripcsak,  Clayton,  Pryor,  and  others,  1990; 
Prokosch,  Kamm,  Wieczorek,  and  Dudeck,  1991). 
Because  development  of  medical  knowledge  bases 
is  so  costly  and  time  consuming,  and  the  benefits  of 
sharing  organized  knowledge  are  so  great,  the 
ASTM  has  created  a  new  subcommittee  on  Health 
Knowledge  Representation. 


Analytic  systems  as  emerging  information  tech- 
nologies include  natural  language  processing,  expert 
systems,  and  transaction  processing  systems. 

Most  provider  documentation  is  text — that  is, 
handwritten,  typewritten,  transcribed  dictation, 
notes  entered  directly  or  through  an  intermediary 
into  an  AMR,  or  electronically  stored  voice  record- 
ings. This  has  made  it  very  difficult  for  QA  and  QI 
professionals  to  locate  desired  findings,  conditions, 
and  actions.  When  text  is  linked  to  coded  data,  iden- 
tification of  relevant  records  is  easier,  but  the  bur- 
den of  manual  text  review  remains  overwhelming. 

Natural  language  processing  (NLP),  the  capacity 
of  automated  systems  to  extract  meaning  from  text 
data,  is  fundamental  to  increasing  our  ability  to 
measure,  assure,  and  improve  health  care  quality. 
NLP  can  be  applied  to  text  for  indexing,  codifying, 
abstracting,  and  interpreting  medical  documenta- 
tion and  information  resources.  There  is  active 
research  and  prototyping  being  done  using  a  variety 
of  analytic  approaches  for  these  functions,  much  of 
it  explicitly  linked  to  QA  and  QI  (Gabrieli  and 
Speth,  1990;  Hersh  and  Greenes,  1990): 

•  Discharge  summaries  are  abstracted  for  asthma 
management  audit  (Lyman,  Sager,  Tick,  and 
others,  1991). 

•  Discharge  summaries  are  audited  for  quality  of 
care  problems  (Gabrieli,  Pollock,  Beltrami,  and 
others,  1989). 

•  Discharge  summaries  are  abstracted  to  create 
patient  profiles  (Borst,  Lyman,  Nhan,  and 
others,  1991). 

•  Discharge  summaries  and  operative  notes  are 
stored  and  indexed  in  a  text  data  base  (Ribitzky, 
Waterman,  Zieman,  and  others,  1990). 

•  Orthopedic  diagnoses  and  procedures  are 
indexed  to  facilitate  clinical  research  (Floyd  and 
Kinser,  1990). 

•  Pathology  reports  are  coded  and  abstracted  to 
SNOMED  controlled  vocabulary  (Moore  and 
Berman,  1991). 

•  Emergency  department  records  are  audited  for 
key  findings  based  on  "trigger  diagnoses" 
(Holbrook  and  Aghababian,  1990). 

•  Admission  summaries  are  abstracted  to  yield 
physical  examination  findings  to  enrich  an 
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existing  clinical  data  base  (Lin,  Lenert,  Middle- 
ton,  and  Shiffman,  1991). 

•  Literature  abstracts  are  indexed  using  a  the- 
saurus (Evans,  Hersh,  Monarch,  and  others, 
1991). 

There  also  are  commercial  NLP  systems  avail- 
able. One  class  of  product  provides  a  user-friendly 
interface  to  relational  data  bases  by  translating  nat- 
ural language  queries  into  the  structured  query  lan- 
guage of  the  data  base.  A  second  class  of  product  is 
provided  with  ASR  systems  to  allow  real-time  cod- 
ing and  interpreting  of  speech  input.  When  coupled 
with  an  expert  system,  quality  review  is  possible 
concurrent  with  automated  data  entry  (Holbrook 
and  Ontai,  1991). 

Expert  systems  were  introduced  earlier  in  this 
paper  and  examples  provided  of  their  incorporation 
into  established,  operational  information  systems. 
Research  and  prototyping  of  more  advanced  expert 
system  applications  and  the  knowledge  bases  sup- 
porting them  flourish,  particularly  in  departmental 
information  systems  and  in  conjunction  with  other 
emerging  information  technologies  (Lincoln,  1990; 
Lipscombe,  1989): 

•  Drug  treatment  of  psychiatric  patients  is  moni- 
tored for  deviations  from  guidelines  or  abnormal 
lab  results  (Bronzino  and  others,  1989). 

•  Selected  clinical  findings  are  abstracted  from  an 
AMR  and  reported  to  a  radiologist  by  an  expert 
consultant  system  to  assist  in  x-ray  interpreta- 
tion (Haug  and  Beesley,  1991). 

•  Diaries  of  cystic  fibrosis  patients  in  a  home  mon- 
itoring program  are  monitored  for  data  quality 
and  health  status  (Slagle,  Finkelstein,  Leung, 
and  others,  1989). 

•  Ambulatory  drug  utilization  is  monitored  with 
reminders  and  chart  audits  (DeNeef,  Ellsworth, 
and  Schneeweiss,  1991). 

•  Insurance  claims  are  monitored  to  identify  read- 
missions  due  to  surgical  complications  (Roos 
and  others,  1985). 

•  Computer-assisted  QA  is  developed  for  a  respi- 
ratory care  program  (Elliot,  1991). 

•  A  PRO's  identification  of  quality  problems  using 
the  UCDS  is  compared  with  an  expert  diagnostic 
system  (Lau  and  Warner,  1991). 


•  Blood  ordering  practices  can  be  modified 
through  the  use  of  a  consultation  system 
(Lepage,  Gardner,  Laub,  and  others,  1991). 

•  Treatment  of  hypertension  patients  in  an  AMR 
setting  is  monitored  and  critiqued  (Van  der  Lei 
and  Musen,  1991). 

•  A  consultative  expert  system  for  cholesterol 
management  uses  a  hypertext  interface 
(Kane,  1990). 

•  A  rheumatological  expert  system  uses  a 
knowledge  base  of  highly  detailed  anatomical 
knowledge  (Horn,  1991). 

Transaction  processing  systems  (TPSs)  are  a 
general  class  of  information  systems  that  monitor 
processes  in  another  operational  system  and  trigger 
specific  actions  if  certain  predefined  events  occur. 
Thus  TPS  provide  real-time  feedback  based  on 
expert  system  rules.  The  term  TPS  is  not  frequently 
used  in  health  care  informatics,  but  it  is  applicable 
to  systems  as  simple  as  error-checking  by  an  intelli- 
gent bedside  terminal  during  interactive  data  entry 
and  as  complex  as  the  evaluation  by  expert  systems 
of  the  pattern  of  codes  produced  by  NLP  of  speech 
data  entry.  In  each  case,  a  transaction  of  data  entry 
is  monitored  and  potentially  interrupted  before 
completion  because  of  the  content  of  the  data. 

Particularly  pertinent  to  provider  data  entry  is 
the  interface  approach  of  context-sensitive  prompts 
and  defaults.  In  this  application,  a  TPS  monitors 
user  input  and  selects  the  most  appropriate  next 
message  for  display  based  on  that  input.  For  exam- 
ple, a  practitioner  entering  a  patient's  complaint  of 
sore  throat  might  be  prompted  to  document  the 
presence  of  coughing,  and  if  present,  its  productive- 
ness. Similarly,  when  prescribing  medications,  the 
provider  may  be  offered  the  most  commonly  used 
dose  and  form  for  the  patient's  age,  subject  to 
replacement  with  other  values.  The  concept  of 
transaction  processing  is  critical  to  improved  data 
quality  and  the  incorporation  of  clinical  guidelines 
into  real-time  documentation  of  patient  care. 

Emerging  data  storage  technologies  of  impor- 
tance to  quality  efforts  include  magnetic  disk  drives 
with  increasing  capacity  and  mean  time  between 
failures  rated  at  over  5  years,  optical  disks,  and 
portable  memory.  Like  magnetic  storage,  the  optical 
disks — where  lasers  are  used  to  permanently  write 
data  onto  minute  segments  of  disk — will  help  meet 
anticipated  needs  for  huge  storage  capacity  as 
AMRs  proliferate  and  integrate  increasing  numbers 
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of  data  sources,  in  particular  ASR-captured  dicta- 
tion and  PACS-captured  images. 

Optical  disc  technology  provides  access  to  refer- 
ence sources  as  well  as  mass  data  storage  (Brahmi, 
1989).  Several  vendors  produce  compact  disc  read- 
only memory  (CD-ROM)  discs  of  bibliographic, 
pharmacologic,  and  textbook  data  bases.  Prototypes 
have  been  developed  that  incorporate  these  data 
sources  into  provider  workstations.  Health  care 
organizations  with  mobile  patient  populations  and 
independent  care  sites  also  are  producing  their  own 
CD-ROMs  to  provide  current,  core  medical  records 
at  all  delivery  sites  (Demetriades  and  Gomez,  1991). 

Portable  memory,  allowing  patients  to  carry 
medical  data  and  make  it  available  to  providers  or 
vendors  unacquainted  with  them,  comes  in  the  form 
of  optical  (laser)  and  integrated  circuit  (smart) 
cards.  Optical  cards  have  a  memory  capacity  of  up 
to  4  megabytes  of  data  and  may  hold  text  and  image 
data  (Bouldin  and  Callen,  1990;  Brown,  1989; 
Gomez,  Demetriades,  Babcock,  and  Peterson,  1991; 
Shiina,  Nishibori,  Matsuzaki,  and  others,  1991). 
They  are  write-once  read-many  (WORM)  cards,  so 
they  are  updated  by  adding  current  data  without 
deleting  prior  data.  Integrated  circuit  cards  contain 
an  electronically-erasable,  programmable,  read- 
only-memory (EEPROM)  microchip  with  a  capac- 
ity of  16  kilobytes  of  data  (Benson,  1990;  Hopkins, 
1990;  Ognibene,  1991;  Regan,  1991;  Smart  cards  in 
Medicine,  1990).  An  essential  medical  record  can  be 
written,  erased,  and  edited. 

Finally,  data  storage  technology  now  supports 
digitization  of  sound.  This  may  be  used  on  a  tempo- 
rary basis  to  make  dictation  available  as  part  of  an 
AMR  until  it  is  transcribed  into  the  record.  In  the 
longer  term,  voice  and  sounds  from  both  providers 
and  patients — including  heart  and  breath  sounds 
and  examples  of  speech  pathologies — may  be  digi- 
tized to  become  permanent  parts  of  the  medical 
record  (McDonald,  1989a). 

Information  processing.  The  core  information 
processing  work  of  QA  and  QI  will  benefit  from 
information  processing  technologies  of  three  types: 
systems  integration,  text  processing,  and  expert  sys- 
tems. Integration  advances  include  data  exchange 
standards,  patient  identification  (ID)  matching, 
developments  in  systems  architecture,  and  high- 
speed network  data  transmission.  Individual  types 
of  exchange  standards  have  been  discussed  above 
and  will  contribute  to  information  processing  for 
QA  and  QI. 


Of  particular  importance  is  the  work  by  the 
National  Library  of  Medicine  (NLM)  to  develop 
the  Unified  Medical  Language  System  (UMLS). 
UMLS  is  designed  to  facilitate  communications 
between  both  different  electronic  systems  and 
different  institutions  by  creating  a  system  for  map- 
ping and  interpretation  between  diverse  coding 
systems  and  controlled  vocabularies  (Cimino  and 
Sideli,  1991;  Humphreys,  Lindberg,  and  Hole,  1991; 
Lindberg  and  Humphreys,  1990;  Masarie  and  others, 
1991;  McDonald,  1991;  Sherertz,  Olson,  Tuttle,  and 
Erlbaum,  1990).  COSTAR,  CPT-4,  DSM-III, 
DXplain,  HELP,  ICD-9,  and  SNOMED  codes 
have  been  included,  as  well  as  the  Medical  Subject 
Headings  (MeSH)  used  by  the  NLM.  It  is  expected 
that  UMLS  will  provide  a  cornerstone  for  NLP  of 
medical  text,  and  it  has  been  incorporated  into  the 
ASTM  data  exchange  standards. 

Closely  related  to  UMLS  is  the  Integrated 
Academic  Information  Management  System,  also 
supported  by  the  NLM.  This  project  involves  a 
number  of  sites  experimenting  with  communica- 
tions and  information  processing  technologies  for 
the  development  of  integrated  health  care  informa- 
tion systems  (Cimino  and  Sengupta,  1991;  Lincoln, 
1990;  Shortliffe  and  others,  1990).  Data  exchange 
and  the  interpretation  of  data  between  systems 
are  the  goals. 

Patient  identifier  matching  is  another  corner- 
stone for  integration  of  information  systems  for  QA 
and  QI.  According  to  AHCPR  (1991,  p.  3-2),  "The 
greatest  technical  impediment  to  linking  personal 
data  files  is  the  lack  of  a  standard  personal  identi- 
fier." Institution  and  system  identifiers  are  not 
coordinated  and  may  change  over  time.  In  many 
nonmedical  information  systems,  the  social  security 
number  has  become  a  de  facto  standard  individual 
identifier.  The  social  security  number  is  being  pro- 
moted for  adoption  in  health  care  by  the  National 
Committee  on  Vital  and  Health  Statistics  of  the 
U.S.  Department  of  Health  and  Human  Services 
(Agency  for  Health  Care  Policy  and  Research, 
1991).  Meanwhile,  institutionally  based  alternatives 
for  patient  data  matching  are  being  studied  (Fair, 
Lalonde,  and  Newcombe,  1991;  Hammond,  Stead, 
and  Straub,  1990;  Sideli  and  Friedman,  1991). 

Classical  systems'  architecture  is  highly  central- 
ized and  based  on  mainframe  computers.  Many 
institutions  are  experimenting  with  and  implement- 
ing other  configurations,  both  distributed  and  based 
on  personal  computers  or  workstations,  and  both 
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interfaced  and  integrated,  to  facilitate  the  process- 
ing and  integration  of  medical  information  (Curtice 
and  Glaser,  1989;  Dayhoff,  Maloney,  and  Kuzmak, 
1990;  Ellis,  1990;  Hammond  and  others,  1990; 
Margulies,  McCallie,  Elkowitz,  and  others,  1990; 
Price  and  others,  1990;  Stead,  Borden,  Boyarsky, 
and  others,  1991;  Tang,  Annevelink,  Fafchamps,  and 
others,  1991;  Young,  Tang,  and  Annevelink,  1991). 

A  final,  closely  related  technologic  advance  is 
the  development  and  diffusion  of  high-speed  data 
communications  through  local  and  regional  net- 
works (Dick  and  Steen,  1991).  Fiber-optics  and  the 
Integrated  Systems  Digital  Network  permit  rapid, 
low-cost  digital  networks  for  voice,  data  and  image 
retrieval  and  transmission  between  disparate, 
decentralized,  and  independent  data  sources  and 
may  permit  the  development  of  a  virtual  AMR. 

Knowledge  dissemination.  Knowledge  dissemi- 
nation, a  key  product  of  QA  and  QI,  also  will  bene- 
fit from  information  systems  integration.  This  is  par- 
ticularly true  in  QI,  where  successive  incremental 
lessons  and  interventions  can  be  implemented  with 
the  aid  of  the  integrated  system,  and  a  new  cycle  of 
monitoring  can  begin.  In  addition,  advanced  deci- 
sion support  and  executive  information  systems  will 
promote  knowledge  dissemination.  Their  success, 
like  that  of  the  AMR,  will  be  linked  closely  to  matu- 
ration of  the  graphic  workstation  with 
intelligent/intuitive  data  display  and  interactive 
feedback  concerning  treatment  plans  via  TPS 
(Hammond  and  Stead,  1991;  Higgins  and  others, 
1991;  Kawahara  and  Jordan,  1989). 

Undeveloped  information  technologies. 
Shortliffe  and  colleagues  (1990,  p.  20)  define 
medical  informatics  (MI)  as  "the  rapidly  growing 
scientific  field  that  deals  with  the  storage,  retrieval, 
and  optimal  use  of  biomedical  information,  data, 
and  knowledge  for  problem  solving  and  decision- 
making." Equating  MI  with  the  term  medical 
information  science,  they  note  that  it  includes  the 
application  of  computers  to  medicine — broadened 
to  include  medical  statistics,  recordkeeping,  and 
the  study  of  the  nature  of  MI — with  a  de-emphasis 
of  the  computer  itself  while  focusing  on  the  nature 
of  the  field  to  which  the  computations  are  applied 
(Shortliffe  and  others,  1990,  p.  19).  Other  authors 
also  have  discussed  this  new  discipline,  including 
those  who  have  adopted  it  as  their  profession 
(Aronow,  Payne,  and  Pincetl,  1991;  Ball  and 
Douglas,  1990;  Braude,  1990;  Lincoln,  1990). 


From  the  field  of  MI  come  underdeveloped  and 
as  yet  undeveloped  systems  and  technologies  that 
may  have  a  significant  effect  on  QA  and  QI.  These 
technologies  include  advances  in  knowledge  repre- 
sentation and  navigation,  theory  concerning  infor- 
mation uncertainty  and  incompleteness,  computer 
modeling,  artificial  neural  networks,  and  data  base 
management  systems. 

Knowledge  representation  in  KBs  is  important 
for  information  exchange  between  automated  sys- 
tems and  for  retrieval  and  comprehension  of  that 
information  (Hersh  and  Greenes,  1990;  Lincoln, 
1990;  Perry,  1990).  Building  on  the  NLP  research 
discussed  earlier,  areas  of  research  include: 

•  Frame-based  representation  (Evans,  Monarch, 
Rothwell,  and  others,  1991;  Masarie  and  others, 
1991). 

•  Belief  networks  (Cousins,  Frisse,  Chen,  and 
Mead,  1991;  Deleu,  Beuscart,  Becquart,  and  oth- 
ers, 1989). 

•  Semantic  networks  (McCray  and  Hole,  1990; 
Rector,  Nowlan,  and  Kay,  1990;  Vries, 
Marshalek,  D'Abarno,  and  others,  1992). 

•  Spatial  knowledge  representation  (Srihari  and 
Zhigang,  1989). 

•  Symbolic  representation  (Wagner,  1991). 

•  Use  of  the  Arden  syntax  (Prokosch  and  others, 
1991). 

•  Representation  through  standardization  of 
semantics  (Galeazzi,  Gangemi,  Rossi  Mori,  and 
others,  1991). 

•  Diagnosis  of  multiple  disorders  through  creation 
of  problem  decompositions  (Wu,  1990). 

Closely  related  to  issues  of  the  representation  of 
knowledge  is  the  problem  of  access  to  and  naviga- 
tion through  relevant  knowledge  (Estey,  Oliver, 
Chueh,  and  others,  1990).  A  main  focus  of  research 
here  has  been  the  use  of  hypermedia,  such  as  hyper- 
text interfaces  to  expert  consultant  systems  for  cho- 
lesterol management  (Kane,  1990),  diagnosis  of  the 
acute  abdomen  (Harvey,  Moodie,  Reda,  and  others, 
1991),  and  the  entry  and  QA  of  interpretive  pathol- 
ogy reports  (Berman,  1991),  as  well  as  the  approach 
of  "knowledge  management"  (Greenes,  Tarabar, 
Krauss,  and  others,  1989). 
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Management  of  uncertainty  has  been  problem- 
atic in  the  development  of  clinical  expert  systems. 
Several  approaches  have  been  studied  at  Stanford 
University  (Shortliffe,  1991).  Further: 

•  Uncertainty  as  a  form  of  entropy  in  information 
systems  has  been  modeled  as  a  measure  of 
physician  attention  to  patient  data  (Gage  and 
Frey,  1991). 

•  Automated  extraction  of  relative  degrees  of 
importance  has  been  used  to  cope  with 
uncertainty  in  expert  systems  (Hudson  and 
Cohen,  1990). 

•  Augmented  decision  tables  have  been  used  to 
manage  probabilistic  data  in  algorithms 
(Shiftman  and  Greenes,  1991). 

•  Probability  tree  software  supporting  random 
parameters  has  been  developed  (Dittus, 
Roberts,  and  Wilson,  1989). 

Computer-based  modeling  allows  for  the  simula- 
tion of  a  system,  process,  or  organization  to  test  its 
performance  under  stressors.  Research  in  this  area 
includes: 

•  Simulation  of  the  effect  of  poor  quality  data  on  a 
QA  screening  data  base  (Pine,  Rogers,  Morgan, 
and  others,  1990). 

•  Modeling  for  intervention  assessment  of  risk 
factor  modification  (Bloom,  1990;  Zhuo, 
Gatewood,  and  Ackerman,  1990). 

•  Use  of  chaotic  models  to  explain  nonlinear 
dynamic  systems  (Cohen,  Hudson, 
Moazamipour,  and  Anderson,  1990). 

Artificial  neural  networks  (ANN)  are  structures 
of  parallel  information  processors  that  can  be 
trained  to  perform  complex  analyses.  Experimenta- 
tion is  underway  to  develop  ANNs  to  interpret 
medical  images  (Piraino,  Amartur,  Richmond,  and 
others,  1991);  detect  malignancy  during  analysis  of 
urine  specimens  (Cohen,  Hudson,  Banda,  and 
Blois,  1991);  diagnose  thyroid  conditions  (Bolinger, 
Hopfensperger,  and  Preston,  1991);  facilitate  speech 
recognition  (Kurogi,  1991);  aid  in  nursing  diagnosis 
(Harvey,  1990);  and  perform  clinical  diagnosis  of 
dementia  (Mulsant,  1990). 

Data  base  management  systems  (DBMSs)  are 
critical  to  efforts  to  store,  integrate,  and  retrieve 
data,  information,  and  knowledge.  Relational 
DBMS  have  become  prominent  in  health  care 


settings  in  the  last  decade  due,  in  part,  to  their 
promises  of  portability  and  flexibility  in  response  to 
changing  needs  of  clinical  managers  (Barsalou, 
Chavez,  and  Wiederhold,  1990;  Friedman,  Hripcsak, 
Johnson,  and  others,  1990;  Stahlhut,  McCallie, 
Waterman,  and  Margulies,  1990).  Their  impact 
on  QA  and  QI  is  not  clear,  although  vendors  are 
marketing  NLP  interfaces  to  relational  DBMSs  to 
facilitate  ad  hoc  query.  Object-oriented  DBMSs  are 
just  beginning  to  be  explored  in  health  care.  As  the 
types  of  data  used  in  the  AMR  and  QA  and  QI 
efforts  continue  to  expand  to  include  usable  text, 
complex  images,  sound,  and  full-motion  video, 
health  care  data  bases  may  need  to  be  distributed 
intentionally  to  optimize  DBMS  characteristics  with 
specific  data  types  (Dick  and  Steen,  1991). 

Newer  technologies,  yet  to  be  tested  in  health 
care  information  systems,  include  holography, 
automated  vision  systems  (Harvey,  DiCaprio, 
Heinemann,  and  others,  1990),  brain  wave  data 
entry  (McDonald,  1990),  and  interactive  video  disc 
presentations  for  patients,  with  computer  recording 
and  analysis  of  their  health  outcome  preferences 
(Patrick,  1991). 

Pragmatic  and  Policy  Issues 

The  most  difficult  issue  facing  the  field  of  infor- 
mation science  is  that  of  protecting  the  privacy  and 
confidentiality  of  data.  Issues  related  to  the  devel- 
opment and  adoption  of  uniform  standards  for 
information  exchange  have  technologic  solutions 
near  at  hand,  and  leadership  in  adopting  a  specific 
set  of  standards  for  MI  systems  at  the  Federal, 
State,  and  professional  society  levels  can  greatly 
accelerate  the  widespread  implementation  of  uni- 
form standards.  The  adoption  of  a  common  patient 
identifier  is  methodologically  and  technically  quite 
simple  but  legally  and  ethically  complex.  Such  an 
action  will  make  the  linkage  of  multiple  data  bases 
at  the  patient  level  a  straightforward  technologic 
procedure  that  can  be  executed  at  any  site  having 
access  to  the  necessary  data  bases.  Likewise,  emerg- 
ing information  technologies  offer  new  capabilities 
to  access  and  manipulate  that  data.  Our  current 
inability  to  safeguard  the  security,  privacy,  and 
confidentiality  of  such  patient  data  stands  out  as 
a  formidable  barrier  to  widespread  adoption  of 
technologic  advances  in  information  processing  to 
support  QA  and  QI. 


69 


Concern  over  widespread  computer  access  to 
private  information  is  growing  among  the  general 
public  and  among  professionals  in  the  information 
technology  field  (Brannigan  and  Beier,  1990; 
Weingarten,  1992).  The  development  of  technologic 
solutions  for  ensuring  data  security  and  confiden- 
tiality is  becoming  a  high  priority,  not  only  in  health 
care,  but  in  many  other  industries  that  collect  and 
process  sensitive  information  (Cobb,  1992).  Recent 
advances  in  information  technology  offer  some 
promising  improvements  in  data  and  systems  secu- 
rity, such  as  call-back  modems  for  remote  dial-up 
access  and  relational  DBMSs  that  allow  controlled 
access  at  the  row,  column,  or  cell  level,  depending 
on  the  user  ID  and  device.  However,  additional 
work  in  this  area  is  needed  if  the  widespread  use 
of  shared  data  bases  for  QA  and  QI  is  to  become  a 
reality  (Agency  for  Health  Care  Policy  and 
Research,  1991;  Dick  and  Steen,  1991;  General 
Accounting  Office,  1991;  Lohr,  Yordy,  and  Thier, 
1988;  Regan,  1991;  Walsh  and  Cortez,  1991). 

Another  issue  is  related  to  the  data  itself.  Two 
sets  of  activities  are  needed:  first,  we  must  acceler- 
ate the  definition  of  uniform  minimum  data  sets, 
and  their  development  and  reporting  by  institutions 
must  be  mandated  by  policy,  as  with  UHDDS,  or 
the  data  bases  necessary  to  support  QA/QI  efforts 
will  not  be  available  for  linkage.  Second,  we  must 
improve  data  quality  through  the  widespread  adop- 
tion of  uniform  coding  practice  standards  and  the 
auditing  of  compliance  with  these  standards.  Roper 
and  Hackbarth  (1988)  stated  that  one  of  HCFA's 
most  important  goals  should  be  to  increase  the 
volume  and  quality  of  information  available  to  the 
public.  However,  many  authors  have  cited  data 
quality  problems  in  currently  available  administra- 
tive data  bases  and  have  called  for  improvements 
in  both  the  kinds  of  data  available  (Blumberg,  1987; 
Chassin  and  others,  1989;  Kahn,  Brook,  Draper,  and 
others,  1988)  and  the  accuracy  and  consistency  of 
coding  practices  (Blumberg,  1988;  Lohr  and  others, 
1988).  Data  quality  is  a  fundamental  concern.  It  will 
be  a  hollow  victory  if  we  succeed  in  addressing  the 
legal,  ethical,  and  technologic  barriers  to  data  base 
linkage  and  access  only  to  find  that  the  resulting 
diamond  is  really  coal. 

Future  Research  Directions 

QA  and  QI  increasingly  require  integration  of 
data  and  information  across  computing  environ- 
ments, delivery  sites,  and  time.  Research  on  the 


legal,  ethical  and  technologic  issues  involved  in  data 
linkage  is  crucial  to  progress  in  this  area.  We  also 
need  to  define  the  degree  to  which  patients  can  and 
should  be  responsible  for  their  own  medical  records 
(to  allow  greater  use  of  portable  records).  Examples 
of  pertinent  research  areas  and  questions  include: 

•  Who  needs  access  to  automated  systems  and  for 
what  purposes?  What  level  of  privacy  protection 
is  acceptable  to  the  general  public? 

•  How  can  information  technology  be  used  to 
provide  acceptable  levels  of  patient  privacy  and 
confidentiality  while  permitting  access  to  clinical 
data  for  legitimate  users? 

•  What  are  the  options  for  common  patient  identi- 
fiers and  how  can  information  technology  facili- 
tate their  incorporation  into  diverse  and  at  times 
parochial  environments? 

•  How  can  information  technology  be  used  to 
create  longitudinal,  multisite,  multisystem  data 
bases  for  care  delivery  and  research? 

•  At  what  site  should  data  integration  occur? 
Should  it  be  based  at  the  user  interface,  the 
physical  system  network,  in  centralized  data 
bases,  or  all  three? 

•  How  can  information  technology  be  used  to 
facilitate  the  integration  of  inpatient  and  out- 
patient data? 

A  second  set  of  questions  involves  our  ability  to 
measure  and  improve  data  quality: 

•  What  is  the  level  of  sensitivity  and  specificity  for 
important  data  elements  in  current  automated 
data  bases? 

•  What  degree  of  variation  in  coding  practices 
exists  across  care  settings  and  caregivers,  and 
what  biases  does  this  variation  introduce? 

•  What  mechanisms  can  be  developed  to  enhance 
and  ensure  coding  consistency  across  institutions 
and  practice  sites? 

•  Is  direct  data  entry  by  practitioners  essential  for 
collecting  data  of  the  quality  needed  to  support 
clinical  QI? 

Additionally,  research  is  needed  on  those  areas 
in  which  emerging  technologies  are  already  being 
applied:  connectivity  and  adoption  and  use  of  CPRs. 
Of  particular  importance  to  connectivity  is  the 
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refinement  and  promulgation  of  uniform  data  sets 
to  permit  collection  and  comparison  of  data  across 
sites,  systems,  and  time.  Essential  to  this  effort  is  the 
refinement  of  technical  standards  and  their  adop- 
tion by  the  health  care  industry.  This  work  should 
be  expanded  to  standardization  of  structures  for 
text  data  and  knowledge  representation.  Examples 
of  pertinent  research  issues  include: 

•  What  is  the  range  of  actual  and  potential  uses 
of  uniform  minimum  data  sets  for  different 
types  of  care? 

•  How  can  we  encourage  the  development  and 
reporting  of  uniform  minimum  data  sets? 

•  What  current  and  additional  data  elements  are 
of  greatest  benefit  to  QA/QI  and  outcomes 
research? 

•  How  can  research  data  sets  be  refined  so  they 
are  moderate  in  size,  contain  easily  accessible, 
well  defined  data  elements,  and  have  clear  and 
focused  intended  uses? 

•  What  definitions  should  be  applied  for  coding 
such  entities  as  "reason  for  visit,"  "chief 
complaint,"  and  "presenting  problem"? 

•  How  can  we  incorporate  UMLS  and/or  other 
data  dictionaries  to  help  standardize  coding? 

•  How  can  the  incorporation  of  UMLS  or  other 
data  dictionaries  support  system  connectivity 
and  quality  of  care? 

•  Should  we  establish  a  uniform  procedure  coding 
system  or  develop  code  mapping  for  linking 
inpatient  and  outpatient  procedure  data? 

•  How  can  information  technology  be  used  to 
standardize  definitions  of  outcomes  and  episodes 
of  care  appropriate  to  different  settings? 

•  How  can  information  technology  be  used  to 
establish  data  source  alternatives  to  claims  data 
for  QA  and  QI,  such  as  dictated  summaries  and 
regulatory  abstracts? 

•  What  is  the  cost/benefit  ratio  of  retrieving  and 
analyzing  free  text  as  opposed  to  controlled 
vocabulary  or  coded  data? 

•  How  can  portable  medical  records  serve  an 
integrative  function? 

Research  into  factors  that  influence  the  adoption 
and  use  of  CPRs  is  vital  to  the  development  and 


promulgation  of  an  important  data  source  to  sup- 
port QA  and  QI  and  improve  the  delivery  of  health 
care.  A  major  factor  for  encouraging  greater  use 
and  accuracy  of  encounter  data  is  its  method  of  cap- 
ture in  the 

CPR.  Data  capture  through  direct  entry  by  providers 
at  the  time  of  the  visit  will  speed  the  improvement 
of  clinical  computing  systems  as  well  as  their  inte- 
gration with  operational  and  financial  systems. 
Practitioner  work  flow  and  information  needs  must 
be  better  understood,  and  efficient,  acceptable 
interfaces  must  be  developed  to  encourage  direct 
data  entry.  As  automated  records  gain  acceptance, 
progress  will  be  needed  in  their  incorporation  of 
expert  systems,  alone  and  in  conjunction  with  trans- 
action processing  systems,  to  improve  the  delivery 
and  monitoring  of  health  care  and  the  dissemination 
of  knowledge  gleaned  through  QA  and  QI  efforts. 
Important  areas  for  further  research  include: 

•  What  are  the  data  and  information  needs  of 
practitioners  that  are  best  met  through  CPRs? 

•  What  apprehensions  do  practitioners  have 
concerning  use  of  automated  systems,  and  how 
can  they  be  overcome? 

•  Are  there  clinical  work  styles  that  differentiate 
medical  specialties  and  affect  systems  implemen- 
tation? 

•  How  can  information  technology  be  used  to 
provide  practice  aids? 

•  Can  demonstration  projects  convince  practition- 
ers of  the  positive  effects  on  quality  and  cost  of 
automated  systems? 

•  Are  there  failed  system  implementations  that  can 
provide  lessons  for  practitioner  engagement? 

•  What  is  the  potential  for  notebook  and  pen- 
based  computing  and  for  automated  speech 
recognition  in  facilitating  direct  data  entry  by 
clinicians? 

•  What  is  the  potential  for  natural  language 
processing  to  facilitate  practitioner  data  entry 
and  conversion  of  practitioner  text  notes  into 
coded  and  searchable  data? 

•  What  is  the  cost/benefit  ratio  of  practitioners 
providing  text  as  opposed  to  controlled  vocabu- 
lary or  coded  data? 
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•  How  can  clinical  practice  guidelines  be  incorpo- 
rated into  automated  systems  in  an  acceptable 
fashion? 

•  How  can  transaction  processing  of  health  care 
interactions  be  implemented  to  provide  real- 
time decision  support? 

•  How  can  expert  systems  support  clinical  practice 
and  QA/QI? 

•  What  is  the  most  cost-effective  design/configura- 
tion for  a  clinician  workstation? 

Another  important  area  for  research  concerns 
the  potential  for  information  technology  to  enhance 
our  ability  to  collect  data  directly  from  patients, 
make  it  available  to  clinicians  at  the  time  when  it  is 
most  useful  for  assisting  their  decisionmaking,  and 
use  it  to  support  QA/QI.  Patients  are  the  principal 
source  of  most  clinical  data  but  have  had  the  least 
involvement  with  automated  medical  information 
systems.  Their  role  as  both  suppliers  and  consumers 
of  health  care  information  is  expanding.  Important 
research  questions  in  this  area  include: 

•  How  can  information  technology  be  used  to 
provide  for  patient  self  education? 

•  How  can  information  technology  be  used  to 
support  more  rapid  triage  of  illness  or  injury, 
particularly  from  sites  outside  of  the  care 
delivery  setting? 

•  How  can  information  technology  be  used  to 
automate  medical  history  collection  and  the 
prompting  of  automated  reminders  addressed 
to  the  patient,  with  or  without  provider 
involvement? 

•  How  can  patient  needs,  preferences,  and  satis- 
faction be  captured,  measured,  and  reported 
using  information  technology? 

•  How  can  information  technology  be  used  to 
collect  data  on  health  care  outcomes  directly 
from  patients?  What  accuracy  and  efficiency 
gains  are  achievable  using  these  technologies? 

Finally,  we  need  to  explore  the  extent  to  which 
information  technologies  can  be  used  to  improve 
our  capabilities  and  efficiency  in  measuring  and 
improving  the  quality  of  health  care,  such  as: 

•  How  can  information  technology  improve 
quality  measurement  and  reporting? 


•  What  aspects  of  care  delivery  are  currently 
measurable  using  automated  data  bases,  and 
how  can  information  technology  be  used  to 
increase  and  diversify  the  measurable  attributes? 

•  How  can  information  technology  be  used  to 
more  accurately  and  efficiently  measure  desired 
attributes  of  care  delivery? 

•  How  can  information  technology  improve  the 
creation  and  application  of  case  mix  systems  and 
severity  of  illness  and  health  risk  indexes  to 
identify  variables  affecting  quality  of  care? 

Improved  knowledge  in  these  areas  will  greatly 
facilitate  QA  and  QI  efforts  to  understand  and 
improve  the  quality  of  health  care. 
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Editors'  note:  The  Agency  for  Health  Care  Policy 
and  Research  commissioned  this  paper  to  provide  a 
forum  for  two  leaders  in  their  respective  sectors  of 
the  health  care  community  to  examine,  from  very  dif- 
ferent perspectives,  the  uses  of  information  in  quality 
improvement  and  quality  assurance.  In  the  first  sec- 
tions of  the  paper,  Dr.  Robert  Panzer  examines  issues 
of  definitions,  indicators,  measurement,  and  applica- 
tions from  the  perspectives  of  physicians  and  institu- 
tions providing  heath  care.  In  the  later  sections,  Carol 
Cronin  examines  issues  surrounding  quality  assess- 
ment and  improvement  from  the  perspective  of  pur- 
chasers of  health  care,  including  employers,  groups 
of  employers,  or  groups  of  employees  who  affiliate  to 
purchase  or  influence  the  delivery  of  health  care. 
Both  authors  raise  a  number  of  issues  and  concerns 
about  the  validity  of  quality  of  care  information  and 
how  this  information  can  be  used  productively.  There 
are,  however,  marked  differences  in  how  these  two 
authors,  and  the  communities  they  represent,  view 
issues  such  as  public  disclosure  of  quality -related 
findings  and  the  potential  costs  and  benefits  of  apply- 
ing less-than-perfect  measures  to  assessment  of 
provider  performance.  Research  on  the  topics  they 
identify  could  shed  considerable  light  on  how  both 
providers  and  purchasers  might  serve  as  agents  for 
quality  assurance  and  quality  improvement. 

Introduction 

In  this  paper  we  use  the  term  "quality  informa- 
tion" to  describe  the  general  concept  of  informa- 
tion used  in  quality  improvement  (QI)  and  quality 
assurance  (QA).  Our  discussion  is  divided  into 
three  sections:  (1)  use  of  quality  information  by 
institutions  and  clinicians  who  are  the  major  suppli- 
ers of  health  care,  (2)  use  of  quality  information  by 
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patients  and  purchasers  who  are  the  major  cus- 
tomers of  health  care,  and  (3)  conclusions  and  iden- 
tified research  needs. 

The  fields  of  QI  and  QA  are  dynamic,  as  is  the 
American  health  care  system.  In  this  paper,  we  pre- 
sent perspectives  based  on  both  knowledge  about 
the  present  and  possibilities  for  the  future.  One 
might  summarize  the  present  as  a  situation  where 
QA  approaches  are  more  prevalent  than  QI 
approaches,  and  where  the  application  of  both  QA 
and  QI  information  is  often  incomplete  or  scientifi- 
cally questionable.  Given  the  very  sparse  current 
knowledge  about  QI  in  health  care,  we  project 
available  information  into  a  future  where  QI  out- 
weighs or  incorporates  QA.  We  also  consider  infor- 
mation use  for  QI  in  the  current  health  care  system. 

Definitions  of  continuous  quality  improvement 
(CQI)  and  total  quality  management  (TQM)  vary. 
For  the  purposes  of  this  paper,  we  use  QI,  CQI,  and 
TQM  interchangeably  to  refer  to  approaches  that 
involve  organizational  transformation,  strategic 
quality  planning,  cross-functional  management,  and 
process  improvement  through  methods  such  as  QI 
teams,  systems  thinking,  use  of  quality  tools,  and 
customer  "mindedness."  We  generally  use  QA  to 
describe  various  processes  of  internal  and  external 
oversight  of  health  care  quality.  In  any  case,  use  of 
information  crosses  all  these  boundaries. 

The  focus  of  CQI  mirrors  the  intent  of  tradi- 
tional QA:  outcomes  (for  example,  products), 
process  (for  example,  systems),  and  structure  (for 
example,  design)  are  all  key  parts  of  the  CQI 
approaches.  However,  in  industrial  QI  translated  to 
health  care,  proven  relationships  between  compo- 
nents of  CQI  and  quality  of  care  are  lacking.  The 
current  approach  to  QI  in  formal  CQI  tends  to 
focus  on  definition  of  the  key  process  or  outcome 
measures  as  part  of  the  activities  of  special  teams 
working  to  improve  a  process  or  resolve  a  problem. 
Information  on  customer  satisfaction  is  of  special 
interest,  and  relevant  measurement  systems  have 
become  increasingly  sophisticated  in  recent  years 
(Nelson,  Hays,  Larson,  and  Batalden,  1989;  Nelson, 
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Landgraf,  Hays,  and  others,  1990;  Nelson,  Larson, 
Davies,  and  others,  1991;  Nelson,  Rubin,  Hays,  and 
Meterko,  1990;  Nelson,  Ware,  and  Batalden,  1990). 
Cross-departmental  quality  activities  analogous  to 
QA  systems  are  also  a  well  defined  part  of  full  CQI 
models  (Malcolm  Baldrige  National  Quality  Award, 
1991). 


Use  of  Ql  and  QA  Information  by  Health 
Care  Providers 

The  use  of  quality  information  by  providers  and 
customers  of  health  care  often  overlaps.  However, 
the  following  sections  emphasize  uses  that  are  pre- 
dominantly provider  oriented,  whether  the  informa- 
tion was  generated  by  the  provider  or  by  an  external 
agency. 

Often,  the  application  of  QA  by  providers  of 
care  is  criticized  for  excessive  focus  on  support 
processes  and  structure  rather  than  actual  clinical 
processes.  QI  efforts,  as  well,  are  criticized  for 
focusing  predominantly  on  support  and  administra- 
tive processes  unrelated  to  patient  care.  There  are 
certain  themes,  however,  that  are  clearly  empha- 
sized more  in  QI  than  in  QA.  For  example,  a  typical 
QA  program  might  assess  patient  access  to  a  service 
by  looking  primarily  at  structural  aspects  such  as 
hours  available  and  staffing  patterns.  A  QI  effort  in 
the  same  area  would  be  more  customer-oriented; 
might  rely  on  patient  satisfaction  surveys  to  meas- 
ure performance  (Cleary  and  McNeil,  1988;  Cleary, 
Keroy,  Karapanos,  and  McMullen,  1989;  Nelson  and 
others,  1989;  Nelson,  Landgraf,  Hays,  Wasson,  and 
others,  1990;  Nelson,  Larson,  Davies,  Gustafson, 
and  others,  1991;  Nelson,  Rubin,  Hays,  and 
Meterko,  1990;  Nelson,  Ware,  and  Batalden,  1990); 
and  would  look  at  the  system  of  making  appoint- 
ments as  a  process  to  improve,  regardless  of 
whether  problems  were  perceived. 

Although  the  QA  model  does  not  explicitly  fos- 
ter it,  many  internal  and  external  QA  programs 
applied  to  providers  are  "bad  apple"  oriented,  with 
a  focus  on  finding  problems  and  someone  to  blame 
(Berwick,  1989).  Incident  reporting  systems  inter- 
nally and  peer  review  organization  (PRO)  activities 
externally  tend  to  have  this  emphasis.  A  QI 
approach  to  identified  problems  would  focus  more 
on  a  search  for  the  process  and  systems  improve- 
ments suggested  by  the  individual  situation. 


Levels  of  Information  Aggregation 

Both  QI  and  QA  information  may  be  aggre- 
gated at  different  levels  for  use  by  providers  of  care: 

•  Individual  incident — for  example,  a  medication 
error. 

•  Patient — for  example,  an  unexpected  death. 

•  Physician — for  example,  increased  adverse  out- 
come rate. 

•  Procedure — for  example,  perforations  during 
endoscopy. 

•  Condition — for  example,  postmyocardial  infarc- 
tion mortality. 

•  Unit — for  example,  intensive  care  unit  severity- 
adjusted  mortality. 

•  Service — for  example,  cardiac  surgery  outcome 
rates. 

•  Institutional — for  example,  composite  report  of 
clinical  indicators. 

•  Multi-institutional — for  example,  comparative 
adjusted  outcomes  at  community,  regional, 
State,  and  national  levels. 

The  statistical  and  clinical  significance  of  appar- 
ent deviations  varies  at  these  levels  of  aggregation, 
as  does  the  target  of  the  analysis. 

Models  of  Quality  Information  Use 

Since  much  of  the  QI  and  QA  activity  nationally 
is  guided  by  the  Joint  Commission  on  Accreditation 
of  Healthcare  Organization  (JCAHO)  standards, 
JCAHO  directions  in  information  use  and  recent 
changes  in  those  directions  are  important  for  formu- 
lating research  questions. 

The  JCAHO 's  quality  standards  have  been  in 
evolution  for  several  years.  In  the  mid-1980s,  an 
"agenda  for  change"  began  with  a  movement  to 
develop  a  core  set  of  clinical  indicators  for  various 
areas  where  data  could  be  collected  in  each  hospital, 
aggregated  nationally,  and  used  both  for  compara- 
tive performance  evaluation  and  by  hospitals  inter- 
nally to  help  them  identify  areas  for  improvement. 
More  recently,  a  QI  orientation  has  been  added  to 
the  JCAHO's  standards  (starting  with  education  of 
senior  management)  without  abandoning  the  clini- 
cal indicators  efforts  (O'Leary,  1991). 
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The  JCAHO  Accreditation  Manual  for  Hospitals 
(1992)  incorporates  a  number  of  revisions  with  the 
stated  purpose  of  simplifying  and  improving  the 
standards  related  to  QA  functions  in  hospitals.  The 
improvements  are  framed  as  ones  that  would  "help 
hospitals  overcome  common  weaknesses  in  current 
QA  practice."  The  weaknesses  described  in  the 
Manual  involving  use  of  quality  information  repre- 
sent areas  for  future  research: 

1.  A  focus  on  clinical  aspects  of  care  alone,  rather 
than  the  full  range  of  "governance,  managerial, 
support,  and  clinical  processes  that  affect  patient 
outcomes." 

2.  Compartmentalization  of  QA  activities  accord- 
ing to  hospital  structure  rather  than  the  flow  of 
patient  care  where  processes  are  "often  cross- 
disciplinary  and  cross-departmental." 

3.  A  focus  on  the  problem  performance  of  individ- 
uals rather  than  how  well  processes  of  care  are 
performed,  coordinated,  integrated,  or  improved. 

4.  Initiating  action  only  for  identified  problems 
rather  than  searching  for  ways  to  improve 
processes. 

5.  Separating  appropriateness  and  effectiveness 
from  efficiency  of  care. 

Comprehensive  QA  Information 
Systems 

QA  program  software.  Numerous  proprietary 
and  shared  software  systems  are  described  here  and 
elswhere  in  this  volume.  The  core  of  many  of  these 
systems  is  a  data  base  for  managing  QA  activities, 
for  example,  incident  tracking,  infection  control, 
transfusion  reaction,  and  medication  error  modules. 
Some  of  these  systems  are  "shells"  where  the  insti- 
tution defines  measures  and  data  collection  mecha- 
nisms, and  the  software  stores  whatever  is  provided. 
Others  add  to  this  a  set  of  prototype  measures,  typi- 
cally built  around  various  clinical  indicators  and 
quality  markers. 

Outcomes-focused  systems.  The  APACHE 
system  (Civetta,  Hudson-Civetta,  and  Nelson,  1990; 
Knaus,  Draper,  Wagner,  and  Zimmerman,  1986), 
MedisGroups  system  (Brewster,  Karlin,  Hyde,  and 
others,  1985),  and  the  Computerized  Severity  Index 
are  examples  of  outcomes  oriented  systems  that 
focus  on  severity-adjustment  of  outcomes  to  guide 
institutional  decisionmaking. 


In  each  case,  comparison  of  observed  with 
expected  outcomes,  adjusted  for  patient  mix  (demo- 
graphics, case  mix,  and  severity),  leads  to  flags  of 
areas  where  performance  deviates  from  the  norm 
(DesHarnais,  1990;  Fink,  Yano,  and  Brook,  1989; 
Flood,  1990).  These  comparative  data  provide  flags 
for  internal  action  but  generally  lack  the  process 
data  of  Ql-oriented  benchmarking  systems. 

The  literature  is  mixed  with  regard  to  the  quality 
of  severity  adjustments  and  particularly  the  validity 
of  determinations  regarding  deviant  patterns 
(Aronow,  1988;  Iezzoni,  1989, 1990a).  For  example, 
questions  have  been  raised  about  the  MedisGroups 
system  regarding  the  accuracy  of  its  severity  adjust- 
ments and  the  validity  of  its  major  morbidity  meas- 
ure (Iezzoni,  1987;  Ash,  Coffman,  Iezzoni,  and 
Moskowitz,  1991;  Iezzoni  and  Moskowitz,  1988). 

Current  internal  uses  of  information.  Many 
reviews  of  internal  uses  of  quality  information 
suggest  that  in  formal  QA  programs  information  is 
often  used  more  to  meet  regulatory  requirements 
than  to  meet  internal  needs  to  improve  care 
processes.  However,  newer  models  of  quality  infor- 
mation use  are  compatible  with  both  QI  and  QA 
approaches,  including  outcomes  management 
(Ellwood,  1988)  and  clinical  indicator  systems  with 
cooperative  data  collection  across  institutions  (e.g., 
Maryland  and  Minnesota  multihospital  clinical 
indicators  projects). 

Use  of  Quality  Information  by  Providers: 
Key  Questions 

One  of  the  central  questions  regarding  the  use  of 
information  by  providers  of  care  is  "which  aspects 
of  the  quality  review  process  are  most  effective  in 
bringing  about  positive  changes  in  practitioner, 
organizational,  or  consumer  behavior?"  The  litera- 
ture provides  little  or  no  experimental  data  to 
answer  this  question.  However,  extrapolation  of 
information  from  the  many  case  studies,  demonstra- 
tions, and  opinion  papers  suggests  some  answers. 
There  appear  to  be  different  themes  for  effective 
internal  and  external  quality  review. 

For  internal  quality  review,  a  nonpunitive  feed- 
back of  meaningful  clinical  information  to  providers 
of  care  is  generally  most  effective,  especially  when 
clinicians  participate  in  the  design  of  the  measures 
and  when  information  is  aggregated  at  the  condition 
level.  This  is  compatible  with  QI,  and  many  of  the 
automated  clinical  information  systems  take  this 
approach. 
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Benchmarking  appears  to  be  most  effective  for 
external  quality  review.  Such  an  effort  seeks  to 
gather  information  about  the  outcomes  of  care  for 
different  providers.  In  QI,  efforts  to  find  these  differ- 
ences are  part  of  a  search  for  the  best  processes  by 
looking  at  those  with  the  best  outcomes.  The  focus  is 
the  opposite  of  many  current  external  reviews  that 
concentrate  on  those  with  the  worst  outcomes.  It  is 
noteworthy  that  many  current  external  review 
systems  start  with  a  stated  emphasis  on  finding  those 
with  better  outcomes,  but  actual  practice  deviates. 

Examples:  Quality  Review  of  CABG 
Surgery 

There  are  a  number  of  published  reports  of 
recent  uses  of  quality  information  in  the  area  of 
coronary  artery  bypass  graft  (CABG)  surgery  that 
illustrate  themes  of  effective  quality  review. 

Northern  New  England  Cardiovascular  Disease 
Study  Group.  The  efforts  of  the  Northern  New 
England  Cardiovascular  Disease  Study  Group 
provide  a  model  for  effective  quality  review 
(O'Connor,  Plume,  Olmstead,  and  others,  1991). 
The  study  group  is  a  research  consortium  with 
voluntary  participation.  Some  would  classify  it  as  an 
example  of  multi-institutional  internal  review,  while 
others  would  see  it  as  a  model  of  external  review. 
Features  of  this  model  consistent  with  the  best 
QI  and  QA  themes  include: 

1.  A  focus  on  patients  with  a  defined  clinical 
condition,  those  undergoing  CABG  surgery. 

2.  A  population  base  with  all  5  cardiac  surgery 
centers  and  all  18  cardiothoracic  surgeons  in  the 
region  participating. 

3.  Prospective  collection  of  data  from  consecutive 

cases. 

4.  Definition  of  data  to  be  collected  through  a  joint 
effort  of  clinicians  and  researchers. 

5.  Nonpunitive  followup  of  differences  found  (for 
example,  surgeons  conducted  a  "round-robin" 
of  visits  to  each  others'  institutions  to  observe 
processes  of  care  and  learn  from  each  other. 

6.  A  heavy  QI  focus,  with  analyses  emphasizing 
volume  effects  and  explanations  for  gender 
differences  in  outcomes  (body  size  rather  than 
quality  related)  instead  of  focusing  on  bad 
providers. 

7.  Hospital  identity  coded  in  all  publications. 


This  group  discovered  that  outcome  differences 
across  institutions  could  not  be  explained  by 
differences  in  case  mix  and  risk  factors,  prompting 
participating  surgeons  and  institutions  to  examine 
and  often  change  their  technqiues  and  systems  of 
care.  Quality  of  care  was  not  found  to  be  the  factor 
resulting  in  worse  outcomes  for  women  than  men 
having  CABG  surgery;  instead,  body  size  was  a 
newly  identified  risk  factor  in  accounting  for 
differences  in  outcomes  between  the  sexes. 

HCFA  mortality  reports.  The  Health  Care 
Financing  Administration  (HCFA)  mortality 
reports,  produced  since  1987,  have  evolved  from 
reporting  case  mix-adjusted  outcomes  for  broad 
categories  of  care  —  for  example,  low-risk  cardiac 
conditions  —  to  specific  diagnostic  or  therapeutic 
categories,  such  as  CABG  surgery  (Hartz,  Krakauer, 
Kuhn,  and  others,  1989).  The  outcome  analyzed  is 
mortality  at  various  intervals  postoperatively  (for 
example,  30,  90, 180  days).  The  reports  are  con- 
strained by  reliance  on  International  Classification 
of  Diseases,  Ninth  Revision,  Clinical  Modification 
(ICD-9-CM)  discharge  diagnoses  and  other  vari- 
ables from  discharge  data  abstracts  (Iezzoni,  1990b). 

HCFA  mortality  reports  are  made  public  within 
a  short  period  after  hospitals  are  notified  of  the 
results.  Hospitals  can  adopt  a  defensive  position 
with  explanations  of  any  deviations,  but  they  do  not 
have  time  to  conduct  meaningful  clinical  investiga- 
tions or  process  improvements.  The  effort  by  hospi- 
tals tends  to  be  focused  on  the  external  review  and 
its  conclusions  about  good  versus  bad  performers. 
In  the  initial  cycles  of  reporting,  results  often  were 
prominently  featured  in  local  and  national  newspa- 
pers with  a  "bad  apples"  approach  to  differences 
between  hospitals,  differences  that  did  not  always 
hold  up  to  clinical  review  and  analysis  of  the  next 
year's  data.  Release  of  hospital  performance  by 
name  is  a  prominent  characteristic  of  these  reports 
that  would  be  avoided  in  a  QI  approach. 

HCFA  mortality  reports  do  use  advanced  multi- 
variate techniques  to  maximize  the  use  of  available 
data  from  the  uniform  hospital  discharge  data  set.  It 
remains  to  be  seen  whether  the  potential  benefits  of 
identifying  outlier  hospitals  and  of  informing  pur- 
chasers can  be  reconciled  to  the  risk  of  misidentify- 
ing  hospitals  that  perform  acceptably. 

New  York  State  cardiac  surgery  analysis. 
Another  analysis  of  CABG  surgery  outcomes 
demonstrates  features  of  effective  quality  review 
and  also  some  of  the  pitfalls  of  external  efforts. 
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The  clinical  variables  of  interest  were  defined  by 
a  State-wide  group  of  active  clinicians  (a  cardiac 
surgery  advisory  committee  that  studies  both 
cardiac  surgery  and  coronary  angioplasty).  Data 
collection  was  prospective  using  a  simple  two-page 
form,  and  staff  of  the  clinicians  involved  performed 
the  actual  data  collection.  Information  was  fed  back 
to  the  clinicians  who  had  onsite  software  available 
to  enter  data  and  analyze  their  own  observed 
and  expected  performances.  Thus  hospitals  had 
the  opportunity  to  examine  outcomes  and  seek 
improvements  if  needed. 

Yet  the  data  released  in  1990  in  conjunction 
with  a  publication  on  the  1989  findings  (Hannan, 
Kilburn,  O'Donnell,  and  others,  1990)  had  many 
of  the  features  of  a  HCFA  mortality  release.  There 
was  much  focus  on  hospitals  with  "bad  perform- 
ance" in  local  and  national  newspapers,  with  hospi- 
tals adopting  primarily  defensive  positions  because 
significant  internal  analysis  and  process  improve- 
ment efforts  were  not  possible  between  the  time  of 
hospital  notification  and  release  of  the  data.  Also, 
several  of  the  hospitals  with  the  worst  performances 
in  the  data  released  in  1990  had  good  results  in  the 
1989  data  and  vice  versa. 

When  contrasted,  most  elements  of  the  North- 
ern New  England  Cardiovascular  Disease  Study 
Group  and  the  New  York  State  cardiac  surgery 
efforts  are  similar,  suggesting  that  both  could  be 
effective  methods  for  improving  outcomes.  How- 
ever, the  New  England  effort  had  more  of  the 
features  commonly  associated  with  QI  approaches 
— for  example,  a  focus  on  joint  examination  of 
processes,  with  cross-hospital  analyses  of  the 
reasons  for  outcome  differences  and  the  processes 
associated  with  those  differences.  The  New  York 
effort  released  the  hospital  name  along  with  the 
performance  data. 

Philadelphia  CABG  analysis.  Another  CABG 
surgery  outcome  analysis  illustrates  the  fine  line 
between  CQI  approaches  and  well-designed 
research  with  feedback.  The  Philadelphia  analysis 
at  five  teaching  hospitals  used  a  retrospective 
method  to  examine  CABG  mortality  rates 
(Williams,  Nash,  and  Goldfarb,  1991).  While  not 
necessarily  developed  as  a  QI  effort,  it  was  a  coop- 
erative study,  and  data  were  shared  with  the  five 
hospitals  whose  outcomes  were  analyzed  well 
before  public  release.  The  results  were  listed 
individually  but  in  coded  form,  unlike  the  HCFA 
and  New  York  State  reports. 


Barriers  to  Improvement 

Other  key  questions  are  "which  methods  have 
health  care  executives  used  to  identify  barriers  to 
improvement,"  and  "which  strategies  have  they 
developed  for  removing  these  barriers?" 

Again,  the  literature  does  not  provide  hard 
evidence,  but  demonstrations,  case  studies,  and 
opinions  suggest  some  general  themes  of  how 
barriers  are  reduced.  The  themes  include  decentral- 
ized decisionmaking,  education  in  CQI  approaches 
and  principles  of  behavior  (Berwick,  1989;  Laffel 
and  Blumenthal,  1989),  and  clinically  robust  quality 
information  approaches  such  as  application  of  out- 
comes measurements  (Tarlov,  Ware,  Greenfield, 
and  others,  1989). 

Organizational  Decisionmaking 

The  question  of  how  quality  information  affects 
organizational  decisionmaking  is  complex  and  more 
an  issue  of  the  management  of  institutions  than  an 
"information  issue."  Themes  from  the  literature 
suggest  four  features  of  quality  information  use  that 
maximize  effects  on  managerial  decisionmaking: 

(1)  integration  of  clinical  and  administrative  efforts, 

(2)  incorporation  of  meaningful  clinical  perform- 
ance (process  and  outcome)  information  into  strate- 
gic planning,  (3)  close  linkages  between  internal 
quality  review  groups  (e.g.,  QA  staff)  and  clinical 
leaders,  and  (4)  nonpunitive,  internally  focused 
analysis  of  processes. 

Time-Oriented  Data 

The  use  of  data  displayed  over  time  has  been 
addressed  in  many  of  the  previous  sections  and 
examples.  For  example,  the  Northern  New  England, 
HCFA,  New  York  State,  and  Philadelphia  analyses 
of  CABG  surgery  outcomes  all  incorporated  time- 
oriented  data  analyses.  Many  of  the  internally 
focused  systems  for  analyzing  severity-adjusted  out- 
comes (APACHE,  MedisGroups)  also  incorporate 
the  routine  ability  to  analyze  performance  over  time 
as  well  as  across  institutions. 

In  typical  QA  and  QI  analyses  of  such  data,  the 
focus  is  generally  similar.  Time-oriented  data  in 
both  QI  and  QA  help  confirm  whether  a  deviation 
is  "real" — that  is,  reproducible  and  not  a  statistical 
artifact — and  followup  data  over  time  help  assess 
the  effectiveness  of  interventions. 
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However,  in  formal  CQI,  use  of  quality  control 
tools  organizes  data  in  somewhat  different  ways. 
Control  charts  organize  data  to  determine  if  the 
occurrence  of  adverse  events  is  random  (due  to 
"common  cause"  problems  where  overall  improve- 
ments in  the  system  lead  to  a  new  level  of  perform- 
ance, regardless  of  the  level  of  performance)  versus 
nonrandom  over  time.  The  latter  are  due  to  "special 
causes";  eliminating  the  special  occurrence  is  the 
first  goal,  and  "tampering"  with  the  overall  system 
is  to  be  avoided.  Furthermore,  performance  within 
expected  bounds  is  not  a  sufficient  goal  in  QI;  the 
QI  philosophy  is  continuous  improvement  until  no 
more  improvements  are  possible. 

Quality  Information  and  Clinical 
Decisionmaking 

A  variety  of  applications  of  quality  information 
directly  affect  clinical  decisionmaking.  They  include 
critical  pathways,  case  maps,  algorithms,  and  case 
management,  where  there  is  organized  effort  to 
"design  in"  a  standardized,  preplanned  set  of  deci- 
sions, as  well  as  information  feedback  to  clinicians 
to  help  them  fine  tune  their  decisionmaking  for 
specified  patient  subgroups. 

Outcomes  management  and  severity  of  illness 
systems  provide  information  to  clinicians  to  guide 
their  choices  more  generally  (O'Connor,  Plume, 
Olmstead,  and  others,  1991),  while  decision  support 
systems  help  them  process  and  assess  complex  infor- 
mation. Most  traditional  QA  information  uses  pro- 
vide feedback  to  clinicians  on  past  decisions,  rather 
than  guide  future  decisionmaking. 

Case  Study:  APACHE  System 

For  a  number  of  reasons,  the  APACHE 
approach  (whether  version  I,  II,  or  III)  is  a  model 
for  use  of  severity-adjusted  outcomes  in  managing 
care  (Knaus,  Draper,  Wagner,  and  Zimmerman, 
1986).  The  core  tool,  severity  adjustment  of  out- 
comes in  a  specific  setting  (the  intensive  care  unit 
— ICU)  with  the  APACHE  scoring  system,  is 
clinically  sensible  and  scientifically  validated. 

The  latest  implementation  through  the  propri- 
etary APACHE  III  ICU  Management  System 
provides  data  collection  and  analysis  systems  to 
"wire"  an  ICU  to  monitor  patient  severity,  expected 
outcomes,  and  therapeutic  needs  on  a  day-to-day 
basis  to  guide  clinical  decisionmaking.  The  intent  is 


that  the  clinician  can  use  the  daily  information  to 
estimate  prognosis  and  to  make  some  of  the  key 
decisions  in  hospital  health  care:  should  this  patient 
continue  to  receive  intensive  care,  is  the  expected 
benefit  of  ICU  care  sufficient,  or  should  another 
patient  be  treated  in  that  bed? 

Time-oriented  analysis  of  the  same  data  in 
comparison  with  expected  outcomes  from  the 
APACHE  mortality  model  also  tells  an  institution 
whether  its  overall  or  condition-specific  performance 
for  patients  receiving  intensive  care  is  on  target. 

Clinical  Information  Systems 

Many  of  the  most  effective  clinical  information 
systems  provide  active  use  of  quality  information 
to  affect  clinical  decisionmaking  (see  the  paper 
by  Coltin  and  Aronow,  in  this  volume).  For  example, 
the  Regenstrief,  TMR,  STOR,  and  COSTAR 
systems  are  comprehensive  ambulatory  care  data  sys- 
tems, with  inpatient  links,  that  can  feed  back  process 
information  to  clinicians  and  affect  their  clinical 
decisions,  through  either  broad  feedback  or  specific 
reminders  (Garrett,  Hammond,  and  Stead,  1986). 

The  HELP  system,  developed  at  LDS  Hospital 
in  Salt  Lake  City,  UT,  is  a  model  for  a  hospital 
information  system  that  directly  affects  clinical 
decisionmaking,  more  through  concurrent  feedback 
on  clinical  actions  than  through  respective  analysis 
of  typical  quality  information. 

Quality  Information  and  TQM 
by  Institutions 

While  many  of  the  potential  uses  of  information 
in  in  health  care  TQM  have  yet  to  be  fully  imple- 
mented, there  are  models  for  use  of  outcomes  and 
process  information  in  the  three  domains  of  TQM: 
quality  planning,  cross-functional  management,  and 
process  improvement  through  individual  and  team 
efforts  (Kuperman,  James,  Jacobson,  and  Gardner, 
1991).  Crossing  these  domains  is  the  use  of  bench- 
marking of  processes  or  outcomes  across  institu- 
tions. The  Malcolm  Baldrige  National  Quality 
Award  (1991)  criteria  for  the  category  of  "Informa- 
tion and  Analysis"  look  very  different  from  existing 
QA-oriented  programs.  The  award  criteria  focus  on 
information  reflecting  the  process  of  care  and  how 
data  are  analyzed  to  promote  the  organization's 
objectives,  rather  than  to  make  judgments  about 
individual  performance  or  problem  areas  alone. 
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The  Baldrige  criteria  explicitly  include  components 
relating  to  information  use  and  emphasize  evidence 
of  actual  use  rather  than  capacity. 

The  literature  to  date  include  no  experimental 
evidence  on  the  effectiveness  or  costs  in  health  care 
of  following  all  or  components  of  the  Baldrige 
Award  criteria  for  information  use.  Case  studies  of 
QI  implementations  in  health  care  focus  on  process 
improvements  and  QI  project  team  efforts,  with  lit- 
tle or  no  reference  to  institutional  QI  information 
systems  and  their  use. 

Use  of  Quality  Information  by 
Purchasers  and  Patients 

Purchasers  of  health  care  have  become  increas- 
ingly interested  in  health  care  quality.  In  a  survey  of 
100  benefits  managers  conducted  for  Metropolitan 
Life  (1990),  over  60  percent  responded  that  health 
care  quality  will  become  a  "somewhat"  or  "much 
more"  significant  issue  over  the  next  2  years.  As  a 
result,  a  majority  of  the  respondents  planned  to 
develop  "a  more  formal  approach  to  managing 
health  care  quality.  A  corroborating  survey  of  250 
hospital  chief  executive  officers  found  that  over  70 
percent  of  the  respondents  indicated  quality  of  care 
will  become  a  key  determinant  of  hospitals'  market 
share  during  the  1990s.  In  addition,  almost  three- 
fourths  said  that  third-party  payers  will  begin  tying 
hospital  reimbursement  to  quality  measures  within 
the  next  5  years  (Johnsson,  1991). 

Intuitively,  cost  and  quality  are  now  related  in 
the  minds  of  many  purchasers,  who  believe  that 
poor  quality  care  is  ultimately  more  expensive  than 
good  quality  care.  Although  this  conclusion  seems 
obvious  and  rather  insignificant,  it  in  fact  represents 
a  major  change  in  attitude  from  the  long  held 
assumption  that  more  health  care,  and  more  expen- 
sive health  care,  equaled  better  health  care.  This 
shift  is  consistent  with  the  themes  of  CQI  and  is  dri- 
ven in  part  by  dissemination  of  CQI  through  large 
segments  of  American  industry. 

The  relationship  between  cost  and  quality  is  cap- 
tured in  the  concept  of  "value",  defined  as  "neces- 
sary health  care  at  reasonable  cost"  (Nash,  1991). 
The  components  of  medical  care  "value  purchasing" 
are  described  as  involving  definitions  of  provider 
cost  and  quality,  measurement  systems  for  provider 
cost  and  quality,  mutually  agreed  upon  standards, 
criteria  for  selecting  providers  based  on  clinical 
performance,  benefit  plan  incentives  to  channel 


employees,  and  reimbursement  adjustments  based 
on  demonstrated  value  (Couch,  1989). 

The  discussion  that  follows  explores  the  poten- 
tial change-agent  role  of  purchasers  in  using  infor- 
mation on  quality  of  health  care.  Three  purchaser 
roles  are  discussed:  (1)  health  care  manager,  which 
covers  the  relationship  between  health  care  pur- 
chasers and  health  care  providers;  (2)  information 
brokers,  which  concerns  the  relationship  between 
health  care  purchasers  and  their  employees,  depen- 
dents, and  retirees;  and  (3)  community  partners, 
which  deals  with  the  relationship  between  health 
care  purchasers  and  their  communities. 

Purchasers  as  Health  Care  Managers 

We  define  a  purchaser  of  health  care  broadly  to 
include  private  employers  who  self-insure  or  pur- 
chase health  care  through  insurers,  labor  unions, 
public  employers,  and  groups  of  employers  who 
affiliate  to  purchase  or  influence  the  delivery  of 
health  care.  Certainly,  the  ability  to  take  on  any  of 
the  following  change  agent  roles  will  depend  on  the 
size  and  resources  of  a  purchaser. 

Pioneering  health  care  research  has  been  con- 
ducted over  the  past  15  years  on  such  quality- 
related  topics  as  small-area  analysis  (Wennberg, 
1990),  appropriateness  of  care  (Brook  and  Lohr, 
1987),  and  the  relationship  between  volume  and 
health  care  outcomes  (Luft,  Hunt,  and  Maerki, 
1987).  The  implications  of  this  type  of  research  have 
resulted  in  the  oft-quoted  observation  by  former 
New  England  Journal  of  Medicine  editor,  Arnold  S. 
Relman  (1991),  that  15-30  percent  of  medical  and 
surgical  procedures  are  not  medically  indicated,  and 
an  almost  equal  number  are  marginally  indicated. 

Continued  escalation  of  overall  and  employer- 
based  health  care  costs  through  the  1980s  ensured 
that  interest  in  these  findings  was  more  than  acade- 
mic, as  purchasers  raised  questions  about  the  value 
and  quality  of  the  care  delivered  to  their  employees, 
dependents,  and  retirees.  In  1991,  total  employer 
health  care  costs  reached  an  average  of  $3,605  per 
employee,  more  than  double  the  amount  5  years 
earlier  (Winslow,  1992).  The  fact  that  the  results  of 
quality  of  care  and  variations  work  began  to  appear 
in  business-oriented  publications  such  as  Business 
and  Health,  Business  Insurance,  and  the  Wall  Street 
Journal  resulted  in  the  transfer  of  knowledge  from 
the  medical  world  to  the  purchasing  world. 
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One  of  the  major  implications  of  research  over 
the  past  several  years  has  been  that  employers 
should  care  about  the  quality  and  value  of  health 
care  provided  to  their  employees.  Previously,  the 
quality  of  health  care  had  not  been  perceived  as  a 
concern.  The  assumption  was  that  purchasers  and 
their  employees  got  what  they  paid  for  without 
much  attention  to  what  they  received  or  how  what 
they  received  related  to  what  they  paid  for.  Recog- 
nition that  health  care  providers  could  be  differen- 
tiated based  on  their  performance,  use  of  resources, 
and  outcomes  led  to  the  emergence  of  "buy  right" 
strategies  that  involved  identifying  the  most  effi- 
cient health  care  providers  in  a  community  and 
offering  incentives  for  employees  to  use  those 
providers  (Traska,  1990).  While  conceptually  clear, 
"buy  right"  is  often  difficult  to  operationalize  given 
the  lack  of  medical  consensus  on  or  measurement 
of  what  constitutes  high  quality,  cost-efficient 
medical  practice. 

A  key  change  in  the  relationship  between  health 
care  purchasers  and  health  care  providers  is  the 
movement  away  from  a  relationship  defined  primar- 
ily through  bill  payment,  to  one  based  on  a  relation- 
ship between  a  customer  (purchaser  and/or  patient) 
and  a  supplier  (health  care  provider).  The  need  for 
supplier  focus  on  customer  satisfaction  based  on 
continuous  evaluation  of  needs  and  expectations 
accompanies  this  shift  toward  the  CQI  model 
(McEachen,  Markens,  Buchanan,  and  Schiff,  1991). 

As  health  care  managers,  purchasers  or  cus- 
tomers have  begun  to  think  in  terms  of  defining 
what  they  are  buying,  gathering  information  and 
making  decisions  based  on  standardized  informa- 
tion, monitoring  health  care  provider  perform- 
ance, and  demanding  accountability  for  results. 
Each  of  these  activities,  the  tools  used  to  support 
them,  and  a  corporate  case  example  are  briefly 
described  below. 

Identifying  health  care  services  to  be  purchased. 
A  first  step  in  the  evolving  formal  customer-supplier 
relationship  is  the  process  of  defining  which  health 
care  services  the  purchaser  wishes  to  buy  in  the 
community.  This  is  usually  done  through  a  process 
of  internal  discussion  involving  a  corporate  benefits 
staff,  corporate  medical  department,  and  often  an 
employee  benefit  consulting  firm.  When  this  process 
is  formalized,  the  result  may  be  a  request  for  pro- 
posal (RFP)  that  is  disseminated  to  community 
health  care  providers  (Miller  and  Miller,  1988). 


RFPs  may  include  requests  for  the  following 
types  of  quality-related  information:  staffing, 
credentialing,  continuing  medical  education,  and 
performance  evaluation  procedures;  continuity  of 
care;  compensation  and  risk-sharing  arrangements; 
utilization  management;  case  management;  devel- 
opment, use,  and  evaluation  of  medical  guidelines; 
descriptions  of  clinical  quality  assessment,  measure- 
ment, or  improvement  programs;  access  to  care; 
and  health  promotion  and  health  education 
(Longo,  Bohr,  Miller,  and  Miller,  1990). 

Other  types  of  information  generated  by  a 
hospital  that  have  been  suggested  to  be  of  value 
to  the  public  include:  nosocomial  infection  rates, 
medication  and  transfusion  errors,  postoperative 
complications,  percent  of  surgical  cases  during 
which  normal  tissue  was  removed,  number  of 
deaths  within  24  hours  of  admission,  and  other 
similar  measures  of  clinical  performance  falling  in 
the  category  of  "quality  markers"  noted  earlier  in 
this  paper  (Hofmann,  1990). 

The  next  generation  of  RFPs  focused  on  identi- 
fying health  care  providers  who  can  treat  a  particu- 
lar set  of  usually  high-cost  and  relatively  infrequent 
medical  conditions.  Often  termed  "centers  of  excel- 
lence," these  are  institutions  with  a  proven  track 
record  for  certain  types  of  procedures  such  as  organ 
transplants,  neurologic  procedures,  and  some  types 
of  rehabilitation.  Companies  such  as  Honeywell, 
Parker  Hannifin,  and  Carlson  Companies  (owners 
of  the  Radisson  Hotel  chain),  as  well  as  many  of 
the  major  insurers,  have  instituted  these  programs. 
The  types  of  quality  of  health  care  data  used  to 
assess  a  potential  center  typically  include  utilization 
data,  death  rates,  morbidity  data,  accreditation 
data,  and  patient  selection  criteria  for  procedures 
(Christensen,  1991). 

Case  example.  The  Rolscreen  Company,  a  self- 
insured,  privately  owned  manufacturer  and  supplier 
of  windows  (Pella),  doors,  sunrooms,  and  skylights 
issued  one  of  the  most  innovative  RFPs  with  respect 
to  quality  of  care  information  (Miller  and  Miller, 
1991).  The  RFP  includes  the  following  statements: 

This  RFP  describes  requirements  for  health 
management  partners.  These  requirements  are 
based  on:  (1)  health  risks  and  health  care  service 
utilization  history  of  employees,  non-Medicare 
retirees,  and  their  dependents;  (2)  predicted  health 
risks  and  health  care  service  requirements  of 
employees,  non-Medicare  retirees,  and  their 
dependents;  (3)  Rolscreen's  goals  and  objectives, 
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that  is  what  the  company  expects  to  accomplish 
with  its  health  management  program;  and  (4) 
how  Rolscreen  hopes  this  initiative  will  positively 
affect  community  residents'  health  and  the  quality 
of  community  health  care. 

The  RFP  discusses  the  responsibilities  of  the 
health  care  manager,  which  include  the  develop- 
ment of  a  system  of  rigorous  clinical  and  other  qual- 
ity criteria  for  each  selective  contracting  service  and 
product  area  (e.g.,  obstetrics,  cardiovascular  care, 
cancer,  primary  care);  negotiation  of  service-specific 
selective  contracting  arrangements  on  behalf  of 
Rolscreen  with  providers  and/or  suppliers  that  meet 
quality  criteria;  and  development  and  implementa- 
tion of  supporting  programs  and  services,  including 
health  risk  assessment,  measurement  of  patient  sat- 
isfaction, management  reporting,  proactive  case 
management,  utilization  management,  and  claims 
administration. 

Supplier  selection/certification.  After  prelimi- 
nary information  is  gathered  and  analyzed  through 
an  RFP  process,  the  purchaser  may  gather  addi- 
tional information  about  the  quality  of  health  care 
prior  to  selection.  This  may  involve  on-site  visits  to 
health  care  providers  to  physically  inspect  the 
premises  and  collection  or  verification  of  informa- 
tion from  government  agencies  such  as  medical 
licensure  boards,  health  departments,  or  insurance 
commissions. 

Increasingly,  private  accrediting  or  standard- 
setting  bodies  such  as  the  JCAHO,  the  National 
Committee  for  Quality  Assurance,  the  Utilization 
Review  Accreditation  Commission,  and  other  pro- 
grams are  positioning  themselves  to  formally  gather 
data  on  the  quality  and  outcomes  of  care  provided 
by  health  care  suppliers.  These  bodies  can  then 
assure  health  care  purchasers  that  certain  standards 
have  been  met  by  suppliers,  thereby  negating  the 
need  for  duplicative  quality  of  care  information 
collection. 

Case  example.  Beginning  January  1, 1990,  Xerox 
Corporation  initiated  Healthlink,  an  employee  ben- 
efits program  that  contracted  with  six  managed  care 
organizations  after  an  exhaustive  review  process 
which  included  questions  on  every  aspect  of  the 
plan's  operations,  oral  presentations  by  competitors, 
and  site  visits.  The  six  companies  serve  as  "network 
coordinators"  and  are  responsible  for  evaluating  the 
performance  of  individual  plans  within  their  net- 
work (Traska,  1990a,  1990b).  With  reference  to 
quality  of  health  care,  Xerox  requires  that  plans  be 


accredited  by  the  National  Committee  for  Quality 
Assurance,  conducts  consumer  satisfaction  surveys, 
and  collects  preliminary  clinical  indicator  quality 
data  in  the  prevention  area  with  respect  to  child- 
hood immunizations  and  cancer  screening. 

Supplier  contracting.  Some  argue  that  guidelines 
governing  a  supplier-customer  relationship  with 
regard  to  the  quality  of  health  care  delivered  should 
be  incorporated  into  the  contractual  language 
between  a  health  plan  and  a  purchaser.  In  arguing 
for  a  pluralistic  model  of  medical  guidelines  devel- 
opment, Havighurst  (1991)  notes  that  such  plural- 
ism allows  private  health  care  plans  to  be  cus- 
tomized to  the  particular  needs  of  the  purchaser. 

Case  example.  As  part  of  a  quality-driven  pur- 
chasing program,  the  Health  Policy  Corporation  of 
Iowa,  an  87-member  coalition  consisting  predomi- 
nantly of  employers  throughout  Iowa,  developed  a 
"recommended  model  agreement"  that  outlined 
model  contractual  specifications  and  accountabili- 
ties for  purchasers  doing  business  with  health  plans 
and  other  suppliers  (Health  Policy  Corporation  of 
Iowa,  1991).  Included  in  the  model  agreement  are 
sections  that  deal  with  the  following  quality  of 
health  care  topics:  data,  information,  and  reporting; 
patient  satisfaction;  quality  of  care;  CQI;  utilization 
management;  case  management;  health  promotion 
and  wellness;  and  provider  performance  objectives. 

Interestingly,  State  legislatures  are  beginning  to 
codify  adherence  to  quality  of  care  guidelines  as  a 
prerequisite  to  supplier  receipt  of  expenditures  for 
public  health  programs.  For  example,  legislation 
passed  in  1991  in  Florida  requires  that  hospitals 
receiving  public  funds  for  maternity  care  must 
specify  the  guidelines  that  are  applied  prior  to  the 
performance  of  certain  cesarean  section  deliveries. 
In  Rhode  Island,  a  legislatively  mandated  commit- 
tee is  responsible  for  recommending  a  set  of 
medical  practice  standards  and  treatment  protocols 
to  be  certified  by  the  legislature  and  used  by 
physicians  as  an  affirmative  defense  in  medical 
malpractice,  by  employers  in  a  value  purchasing 
process,  and  by  State  legislators  as  a  possible  means 
of  determining  reimbursement  policies  for  the  State 
Medicaid  program  (Quality  Resource  Center,  1992a, 
1992b,  1992c). 

Supplier  performance  monitoring/accountability. 
Once  the  contract  between  customer  and  supplier  is 
established,  the  relationship  moves  into  one  of 
monitoring  performance  and  measuring  account- 
ability against  standards.  As  described  earlier, 
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baseline  information  is  often  collected  in  the  RFP 
process.  Information  collection  related  to  the 
quality  of  care  delivered  could  include  such  topics 
as  the  frequency  and/or  length  of  stay  for  proce- 
dures that  have  a  wide  variation  nationally  (e.g., 
laminectomy,  hysterectomy,  CABG  surgery, 
cesarean  sections);  adverse  event  occurrences  that 
may  indicate  compromised  care  (hospital  readmis- 
sions,  premature  deliveries,  and  so  forth);  and 
indicators  related  to  preventive  care  (immunization, 
mammograms,  pap  smears).  In  addition,  purchasers 
are  interested  in  the  longer  term  functional  out- 
comes of  care  delivered  by  suppliers.  Under  the 
auspices  of  the  Managed  Health  Care  Association, 
employers  are  working  with  health  plans  to  collect 
and  pool  outcomes-related  data  on  angina,  asthma, 
and  low  back  pain.  This  will  allow  purchasers  to  fol- 
low individuals  over  time  to  learn  who  benefits  from 
treatment  and  what  costs  and  qualitative  differences 
exist  among  treatment  settings  (Interstudy,  1992). 

Purchasers  also  are  interested  in  the  satisfaction 
of  their  employees,  dependents,  and  retirees  regard- 
ing interaction  with  health  care  providers.  Propo- 
nents of  involving  consumers  in  judging  quality  of 
care  make  the  following  arguments:  (1)  consumers' 
assessments  of  quality  predict  their  behavior  in  the 
market;  (2)  consumers  need  not  be  clinical  experts 
to  provide  accurate  information  about  quality; 
(3)  data  from  patients  are  less  expensive  than 
other  data  sources  (medical  record  audits,  data 
base  searches);  and  (4)  data  from  consumers 
provide  information  not  otherwise  available 
(Davies  and  Ware,  1988). 

Case  example.  The  Bay  Area  Business  Group  on 
Health,  a  coalition  of  large  employers  in  the  San 
Francisco  area,  conducted  an  employee  medical  plan 
satisfaction  survey  on  behalf  of  six  member  compa- 
nies in  1990.  A  total  of  163  health  plans  nationwide 
were  evaluated  by  14,437  respondents  representing  a 
46  percent  response  rate.  On  a  plan-by-plan  basis, 
the  study  revealed  dramatic  differences  with  respect 
to  employee  satisfaction.  The  plan  with  the  highest 
rating  in  the  area  of  physician  care  had  97  percent 
satisfied  customers;  fewer  than  50  percent  of  the 
respondents  in  the  lowest  rated  plan  were  satisfied 
with  physician  care.  When  asked  about  how  likely 
they  were  to  switch  to  another  plan,  employee's 
responses  ranged  from  a  high  of  nearly  40  percent 
who  were  likely  to  switch  to  a  low  of  4  percent  (Bay 
Area  Business  Group  on  Health,  1991). 


Building  long-term  partnerships.  Although 
implied  in  many  of  the  roles  described  in  preceding 
sections,  particular  mention  should  be  made  of  pur- 
chaser interest  in  building  long-term  relationships 
with  a  relatively  small  number  of  suppliers  based  on 
cost-effectiveness  and  quality.  Many  purchasers 
have  been  overwhelmed  by  the  explosion  of  ven- 
dors selling  services  for  every  aspect  of  health  care. 
Others  reduce  the  number  of  suppliers  as  part  of 
their  TQM  efforts. 

Case  example.  Bull  Information  Systems,  a  high 
technology  computer  company,  and  Harvard 
Community  Health  Plan,  a  Boston-based  staff  and 
group  health  maintenance  organization  (HMO),  are 
collaborating  on  a  long-term  partnership  project 
with  a  focus  on  reducing  the  employee's  risks  for  car- 
diovascular and  other  lifestyle-affected  health  prob- 
lems. The  project  will  involve  several  steps  including 
a  health  risk  appraisal  administered  to  all  employ- 
ees, analysis  of  the  data  to  identify  problem  areas 
related  to  lifestyle  factors,  a  report  on  suggested 
interventions  to  improve  health  outcomes  in  those 
areas,  and  implementation  of  the  interventions 
beginning  in  1993  (Quality  Resource  Center,  1992a). 

Information  Brokers 

Second,  purchasers  may  assume  a  major  role 
as  brokers  of  information  between  suppliers  and 
consumers  of  care  (Friedman,  1992).  Building  on 
past  efforts  involving  worksite  wellness  and  health 
prevention  and  education,  purchasers  are  increas- 
ingly interested  in  arming  their  employees  with 
information  that  can  be  used  in  decisionmaking 
about  health  care  providers  or  alternative  treat- 
ment options. 

The  role  of  information  broker  can  take  many 
forms.  In  some  cases,  the  purchaser  may  simply 
bring  information  to  the  attention  of  employees 
through  consumer-oriented  newsletters  they 
purchase  from  a  growing  number  of  health  care 
information  vendors  (Quality  Resource  Center 
Update,  1991).  Other  corporate  communication 
vehicles  include  posters,  bulletin  boards,  company 
newspapers,  videos,  kiosks,  computer  programs, 
mailings,  closed  circuit  television,  or  face-to-face 
meetings  (Ailes,  1991). 

Corporate  information  can  be  either  generic 
(information  about  hospitals,  HMOs,  or  physicians) 
or  condition-specific.  Over  30  States  have  mandated 
the  development  of  public  domain  health  data  sets, 
with  many  of  those  requiring  hospitals  to  submit  an 
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abstract  for  each  discharge  from  their  facility  (Terry 
and  Epstein,  1992).  For  example,  the  Quaker  Oats 
Company  has  used  data  from  the  Illinois  Health 
Care  Cost  Containment  Board  and  repackaged  it 
into  a  series  of  booklets  comparing  hospital  prices 
on  selected  inpatient  and  outpatient  procedures 
(Quaker  Oats  Company,  1990).  Given  that  most 
State  data  collection  efforts  currently  focus  on 
financial  and  utilization  information,  most  public 
domain  data  have  not  yet  included  information 
about  the  quality  of  health  care.  As  States  incorpo- 
rate quality  measurement  systems  into  their  collec- 
tion efforts,  this  type  of  information  will  be  released 
to  health  care  information  brokers. 

In  other  cases,  the  purchaser  may  participate 
in  or  design  an  information  campaign  targeting 
employees,  dependents,  and  retirees  around  a  spe- 
cific medical  condition  or  diagnosis  such  as  prenatal 
care  (Cronin  and  Hartman,  1989),  depression 
(Vaccaro,  1991),  cesarean  sections  (Cronin  and 
Koepke,  1991a),  or  alcohol  abuse  (Koepke,  1991). 

Special  mention  should  be  made  of  the  role  of 
health  care  coalitions  as  health  care  information 
brokers.  For  example,  the  Health  Care  Purchasers 
Association  of  Puget  Sound  (1991)  published  a  32- 
page  consumer  guide  that  includes  sections  on 
doctors,  hospitals,  HMOs ,  preferred  provider 
organizations,  medical  quality,  surgery,  emergency 
care,  maternity  care,  diagnostic  tests  and  X-rays, 
medications,  the  cost  of  care,  and  employee  bene- 
fits. The  Houston  Area  Health  Care  Coalition  has 
published  consumer-oriented  guides  on  mental 
health  and  maternity  care.  The  Pennsylvania  Health 
Care  Cost  Containment  Commission  publication  of 
hospital  performance  data  was  prompted  by 
business  interests  in  "buy  right"  health  care  purchas- 
ing, though  the  commission  itself  is  legislatively 
mandated  (Shanahan  1991).  Finally,  coalitions  in 
Florida,  St.  Louis,  and  Pennsylvania  have  expressed 
interest  in  Health  Pages,  a  consumer  oriented 
magazine  that  provides  price  and  service  informa- 
tion on  doctors,  hospitals,  managed  care  plans, 
nursing  homes,  and  other  medical  professionals  in 
those  areas  (Schneider,  1991). 

Community  Partners 

A  third  change  agent  role  that  purchasers  can 
play  with  regard  to  health  care  quality  is  at  the  com- 
munity level.  Increasingly  purchasers  are  viewing 
their  relationships  with  suppliers  not  only  in  the 
context  of  contractual  relationships  but  also  in 


terms  of  the  need  to  improve  the  health  of  the 
community.  Partnering  in  this  regard  can  take  a 
number  of  different  approaches. 

Community-level  aggregate  data  gathering. 
Several  health  care  coalitions  are  exploring  the 
design  and  implementation  of  a  community  health 
management  information  system.  Using  the  same 
technology  found  in  the  financial  services  industry 
(for  example,  bank  teller  machines,  credit  cards) 
and  grant  monies  from  the  John  A.  Hartford  Foun- 
dation, coalitions  in  Iowa,  Memphis,  and  Seattle  are 
designing  business  plans  that  will  involve  specifica- 
tions for  both  a  community-level  data  base  (patient 
profiles,  provider  and  employer  profiles,  community 
level  health  statistics,  and  so  forth)  and  a  transac- 
tion processing  role  (eligibility  analysis  for  health 
care  providers  to  establish  patient  enrollment  status, 
available  coverage,  and  utilization  review  require- 
ments) (Quality  Resource  Center,  1992a). 

Improving  community  focus  on  health  quality 
problems.  The  National  Partnership  to  Improve 
Maternal  and  Child  Health,  convenes  local  leaders 
in  commununities  nationwide  to  create  dialog 
around  problems  and  solutions  related  to  maternal 
and  child  health.  The  meetings  have  brought 
together  community  leaders,  who  previously  had 
rarely  communicated,  to  achieve  commonly 
accepted  goals  (Bacon,  1991) 

Community-level  CQI.  In  Kingsport,  TN, 
Milwaukee,  and  Chicago,  hospital  and  business 
leaders  have  joined  forces  to  apply  QI  principles  to 
such  issues  as  coordinating  care,  managing  back 
pain  cases,  reducing  hospital  readmissions,  and 
reducing  claims  denials  by  utilization  review  firms 
(Anderson,  1991). 

Public  Policy  Initiatives 

Employers  and  health  care  coalitions  have  been 
at  the  forefront  of  efforts  to  collect  and  analyze 
hospital  and  physician  data  by  State  health  care 
data  organizations.  In  addition,  national  employer 
organizations  such  as  the  National  Association  of 
Manufacturers  and  the  Washington  Business  Group 
on  Health  have  been  supportive  of  increased  public 
funding  for  research  on  QI  and  measurement. 

One  should  not  overestimate  the  simplicity  or 
likelihood  of  success  of  relationships  at  the  commu- 
nity level.  In  some  communities,  the  same  types  of 
initiatives  listed  above  lead  to  divisiveness,  conflict, 
or  lack  of  interest. 
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Patients  as  Agents  for  Ql  and  QA 

The  physician-patient  relationship  is  ultimately 
the  heart  of  the  health  care  delivery  system.  The 
relationship  takes  on  mythic  proportions  that  have 
been  described  as  a  "sacred  trust"  (Friedman,  1992). 
Indeed  most  arguments  for  or  against  doing  or  not 
doing  almost  anything  in  medicine  are  made  on 
behalf  of  the  patient. 

The  etymology  of  the  words  patient  and  con- 
sumer is  relevant.  Webster's  New  World  Dictionary 
of  American  English  (1990)  defines  patient  as  "One 
who  or  that  which  is  passively  affected;  a  person 
who  is  under  medical  treatment."  The  word  is 
derived  from  the  Latin  pad,  which  means  to  suffer. 
In  contrast,  "consumer"  is  defined  as  "the  buyer  or 
user  of  commodities  and  services,"  and  its  Latin 
derivative,  sumere,  means  to  take  up.  Having  been 
socialized  to  be  "passively  affected,"  do  patients 
have  the  capacity  and  interest  to  "take  up"  informa- 
tion about  the  health  care  system  and  be  agents  of 
change  regarding  the  quality  of  care  delivered  to 
them  and  their  families? 

A  review  of  recent  survey  results  with  regard  to 
patients'  views  on  the  quality  of  health  care  they 
receive  indicates  that  most  people  are  satisfied  with 
the  quality  of  their  own  health  care  but  perceive 
that  the  overall  health  care  system  is  in  turmoil 
(Employee  Benefits  Research  Institute,  1991; 
Mellman,  1992). 

Choosing  a  Health  Care  Provider 

How  do  consumers  choose  their  health  care 
providers?  A  review  of  the  literature  indicates  that 
relatively  little  research  has  been  done  on  this  issue. 
Patient  intuition  and  subjective  assessment  based  on 
such  factors  as  personal  impressions,  proximity,  and 
limited  knowledge  about  their  performance  out- 
comes are  often  cited  as  the  criteria  underlying 
physician  selection  (Daley,  Gertman,  and  Delbanco, 
1988).  Steiber  (1988)  reported  the  results  of  a 
national  poll  conducted  for  Hospitals  magazine,  in 
which  consumer  satisfaction  was  found  to  be  influ- 
enced more  by  concern  shown  for  the  patient  than 
by  clinical  care.  For  example,  cleanliness  and  visit- 
ing policies  were  two  factors  that  patients  rate  at  the 
highest  satisfaction  levels.  In  a  survey  of  1,000 
employees,  a  doctor's  reputation  was  the  chief  crite- 
rion in  the  selection  of  a  physician  (Managed  Care: 
The  Employee  Perspective,  1991). 


Given  that  there  is  little  or  no  information  about 
physicians  available  to  the  average  patient,  it  is  not 
surprising  that  informal  and  nonscientific  selection 
criteria  are  used  by  patients.  In  contrast,  in  recent 
years,  limited  information  has  been  made  available 
to  the  public  about  hospital  quality  measured  in 
terms  of  hospital  volume  and,  in  some  cases,  patient 
outcomes.  As  described  earlier  in  this  paper,  since 
1987  data  have  been  released  by  the  Federal  Gov- 
ernment regarding  hospital  mortality  rates  for 
Medicare  patients  with  selected  diagnoses  (Health 
Care  Financing  Administration,  1987). 

Citing  the  potential  for  use  of  this  type  of  infor- 
mation in  patients'  choice  of  hospital,  researchers 
have  estimated  the  influence  of  quality,  charges, 
ownership,  or  distance  on  the  choice  of  hospital  for 
patients  with  seven  surgical  procedures  and  five 
medical  diagnoses  in  hospitals  in  California  in  1983 
(Luft,  Garnick,  Mark,  and  others,  1990).  Though 
the  research  was  conducted  prior  to  the  release  of 
much  public  data,  the  authors  were  interested  in 
whether  hospital  choice  decisions  were  already 
"quality-sensitive"  and  therefore  more  readily  able 
to  incorporate  newly  emerging  public  data.  They 
found  that  hospital  quality  does  play  an  important 
role  in  hospital  choice.  They  assume,  however,  that 
this  reflects  primarily  physician  referral  patterns, 
perhaps  influenced  by  patient  concerns  about  differ- 
ences in  quality  across  hospitals.  They  conclude  that 
rather  than  focusing  on  the  public  to  use  complex 
information  effectively,  detailed  quality  of  care 
information  should  be  made  available  to  physicians, 
while  patients  should  be  made  aware  that  quality 
may  vary  substantially  between  hospitals. 

Despite  the  availability  of  several  years  of  Fed- 
eral and  State  public  domain  health  information, 
there  is  no  known  research  evaluating  its  direct  use- 
fulness to  consumers  of  health  care. 

The  Congressional  Office  of  Technology  Assess- 
ment released  a  landmark  study  in  1988  evaluating 
the  reliability  and  validity  of  indicators  of  medical 
care  quality  that  consumers  could  use  to  choose 
physicians  and  hospitals  (Office  of  Technology 
Assessment,  1988)  The  10  indicators  included 
hospital  mortality  rates,  adverse  events  that  affect 
patients  (e.g.,  nosocomial  infections),  formal  disci- 
plinary actions  taken  against  physicians  by  State 
medical  boards,  PRO-recommended  sanctions, 
malpractice  compensation,  evaluation  of  physician's 
performance  in  caring  for  a  particular  condition  (for 
example,  hypertension  screening  and  management), 
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volume  of  hospital  and  physician  services,  scope 
of  hospital  services,  physician  specialization,  and 
patients'  assessments  of  their  care.  In  a  1990  sum- 
mary of  their  findings,  the  authors  conclude  that 
none  of  the  indicators  examined  provides  definitive 
information  on  the  quality  of  a  hospital  or  physician 
across  the  range  of  medical  care  (Sisk,  Dougherty, 
Ehrenhalft,  and  others,  1990).  They  also  made  the 
following  three  recommendations: 

•  Organizations  and  individuals  can  use  some  of 
the  indicators  as  screens  to  flag  areas  that  merit 
further  examination. 

•  Information  about  unacceptable  care  justifies 
more  attention  than  information  that  ranks  good 
quality  providers  because  of  the  greater  risks 
posed  by  poor  quality  and  the  current  state  of 
QA  techniques. 

•  Consumers  would  be  well  advised  to  combine 
information  from  more  that  one  indicator  and 
for  more  than  1  year  to  increase  the  likelihood 
that  the  information  is  accurate. 

Interestingly,  the  popular  media  are  increasingly 
targeting  information  to  health  care  consumers.  The 
number  of  consumer-oriented  health  care  columns 
or  newspaper  or  magazine  sections  on  health  has 
increased  dramatically  in  the  last  10  years.  One 
trend  is  that  of  regional  magazines  conducting  sur- 
veys of  physicians  or  other  health  care  providers 
and  asking  them  whom  they  would  recommend. 
Survey  results  published  in  The  Washingtonian 
magazine  are  typical  (Stevens,  1991).  Over  3,000 
physicians  in  the  Washington-Baltimore  area  were 
asked  to  name  specialists  and  hospitals  to  whom 
they  would  send  members  of  their  families  for 
particular  kinds  of  treatment.  Published  results 
from  the  825  physicians  who  responded  included 
the  names  of  580  "top  Washington  physicians" 
arranged  by  specialty  area. 

Ongoing  Evaluation  of  Health  Care 
Providers 

If  information  on  consumer  choice  of  health  care 
providers  is  limited,  even  less  is  known  about  how 
consumers  evaluate  their  care  on  an  ongoing  basis 
and  how,  when,  and  why  they  decide  to  switch 
providers.  The  lack  of  knowledge  is  particularly 
acute  in  traditional  fee-for-service  medicine,  where 
consumers  have  total  choice  in  whom  they  see  and 


how  often.  Given  the  lack  of  feedback  loops 
between  patients,  physicians,  and  hospitals,  there 
are  perhaps  only  a  small  number  of  ways  traditional 
fee-for-service  health  care  providers  know  how  their 
performance  has  been  evaluated  by  patients: 

•  The  patient  leaves  the  provider.  In  many  cases, 
the  provider  may  not  even  know  that  a  patient 
has  left  the  practice  given  the  lack  of  process  and 
systems  to  "terminate"  a  relationship.  Requests 
for  copies  of  medical  records  may  be  one  indica- 
tion, change  of  address  notifications  another.  In 
most  cases,  the  provider  probably  does  not  know 
why  the  patient  has  left  the  practice,  thus  negat- 
ing the  opportunity  for  learning. 

•  The  patient  tells  the  provider  he  or  she  is  leaving 
the  practice  or  choosing  another  hospital  and  the 
reasons  why. 

•  The  patient  sues  the  physician  or  complains  to  a 
State  medical  licensing  board,  a  peer  review 
organization,  or  other  public  agency. 

As  discussed  above  with  reference  to  health  care 
purchasers,  health  care  providers  also  are  discover- 
ing the  importance  of  viewing  their  patients  as 
"customers."  Market  research  on  who  one's  patients 
are,  where  they  come  from,  why  they  come  to  a 
particular  provider,  how  the  provider  ranks  among 
competing  providers,  and  how  much  of  the  market 
a  provider  controls  are  all  factors  that  could  be 
pursued  (Omachonu,  1990). 

Increasingly,  consumers  are  finding  themselves 
in  managed  care  health  plans.  While  difficult  to 
describe  succinctly,  most  managed  care  integrates 
the  financing  and  delivery  of  health  care  services 
and  includes  the  following  four  basic  elements: 
(1)  arrangements  with  selected  providers  to  furnish 
comprehensive  benefits;  (2)  selection  standards  for 
providers;  (3)  formal  programs  for  ongoing  QA 
and  utilization  review;  and  (4)  significant  financial 
incentives  for  members  to  use  providers  and 
procedures  associated  with  the  plan  (Hoy,  Curtis, 
and  Rice,  1991). 

Given  that  individuals  enrolled  in  managed  care 
can  vote  with  their  feet  regarding  which  providers 
they  choose  and  when  they  choose  them,  they  offer 
a  natural  setting  for  evaluating  health  care  con- 
sumer choices.  The  fact  that  the  plans  usually  cap- 
ture some  level  of  detail  about  the  patient-plan- 
physician  encounter  enables  them  to  better  target  a 
population  for  studies  on  such  issues  as: 
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•  Enrollment/ disenrollment.  Why  do  consumers 
enroll  in  plans?  Why  do  they  leave? 

•  Use  of  providers  in-network  vs.  out-of-network 
or  switching  health  care  providers  in-network. 
How  do  consumers  evaluate  when  to  use  a  plan 
sponsor  and  when  to  go  out  of  plan?  How  do 
consumers  choose  providers  within  a  plan?  Why 
do  they  switch  to  another  provider  within  a  plan? 

•  Utilization  review.  Under  most  managed  care 
arrangements,  the  physician  is  required  to  con- 
tact an  intermediary  regarding  the  treatment 
plan  for  the  patient.  How  do  patients  respond 
to  a  variety  of  utilization  review  requirements? 
Does  it  affect  physician-switching  behavior?  Are 
patients  more  informed  about  their  treatment 
alternatives  after  a  utilization  review  process? 

•  Consumer  satisfaction  surveys.  As  previously 
mentioned,  plan  members  can  be  directly 
queried  about  their  satisfaction  with  an  overall 
plan  and  its  individual  physicians  and  other 
providers. 

Participation  in  Community-Wide 
Activities 

In  addition  to  consumers  as  quality  of  care 
change  agents  in  their  choice  and  ongoing  evaluation 
of  providers,  their  participation  in  community-wide 
activities  can  provide  them  access  to  health  care 
policy  decisionmaking  regarding  the  quality  of  care. 
As  previously  discussed  with  regard  to  health  care 
purchasers,  consumer  participation  in  health  care 
related  boards,  committees,  and  other  relevant  orga- 
nizations can  bring  the  perspective  of  the  ultimate 
end-user:  the  patient.  Organizations  such  as  the 
American  Association  of  Retired  Persons  (AARP) 
and  others  have  developed  programs  to  help  train 
consumers  to  fulfill  this  role.  For  example,  AARP 
sponsors  the  Citizen  Advocacy  Center,  which  is  a 
support  center  for  public  members  of  health  care 
regulatory  and  governing  boards  such  as  Medicare 
Peer  Review  Organizations  and  medical,  nursing, 
and  nursing  home  administrator  licensing  boards. 

Information  Use  by  Purchasers  and 
Patients 

There  is,  and  will  continue  to  be,  tension 
between  customers  and  suppliers  over  whether  to 
"act  now"  using  quality  of  health  care  information 
that  is  less  than  perfect  and  may  be  clinically 
questionable  or  to  wait  to  act  until  more  meaningful 


data  are  available.  The  reality  of  an  increasingly 
informed  and  involved  purchaser  and  consumer 
community  dictates  that  "no  action"  is  probably  not 
an  option.  There  also  seems  to  be  a  customer  will- 
ingness to  work  with  the  supplier  community  (and 
perhaps  the  research  community)  in  moving  the 
development  and  use  of  quality  information  for- 
ward, perhaps  by  applying  the  principles  of  CQI  to 
the  process  itself.  Given  the  newness  of  many  of 
these  activities,  there  is  an  opportunity  for  the 
Agency  for  Health  Care  Policy  and  Research  to 
fund  research  into  much  of  what  is  being  tried  by 
purchasers  and  groups  of  purchasers. 

Many  of  the  role  changes  described  in  preceding 
sections  of  this  paper  assume  a  fundamental  change 
in  the  practice  of  medicine  with  regard  to  the 
quantity  of  information  made  available  to  measure 
performance.  One  cannot  underestimate  the  degree 
of  corporate  culture  and  behavior  change  this  will 
entail,  which  will  engender  resistance,  efforts  to 
co-opt,  lawsuits,  and  politics.  This  also  suggests  that 
research  is  needed  to  enhance  understanding  of 
cultural  and  behavioral  change  on  the  part  of  health 
care  providers  which  may  involve  the  disciplines  of 
sociology,  psychology,  organizational  development, 
anthropology,  and  communications. 

At  the  same  time,  an  equally  fundamental 
change  is  assumed  in  the  role  of  "purchasers"  from 
"bill  payers"  to  "health  care  managers"  and  from 
"patients"  to  "consumers."  It  is  not  yet  clear  what 
the  parameters  of  these  roles  will  be  or  how  much 
resistance  there  will  be  to  assuming  them.  There  is 
a  certain  comfort  in  not  being  responsible  for  one's 
health  or  the  health  of  others.  In  addition  to  the  dis- 
ciplines mentioned  previously,  research  that  borrows 
from  the  field  of  business  administration  may  be 
helpful  (product  design,  marketing,  organizational 
psychology,  customer  satisfaction,  and  so  forth). 

The  employer  community  is  beginning  to  define 
a  vision  of  the  future  health  care  delivery  system  it 
desires.  Termed  "organized  systems  of  care,"  the 
system  represents  the  next  generation  of  managed 
care.  While  the  components  of  the  delivery  system 
are  still  being  developed,  in-house  systems  of  qual- 
ity control  and  external  demands  for  accountability 
will  be  fundamental  features.  Some  of  the  elements 
of  these  systems  are  incorporated  into  current  state- 
of-the-art  purchaser  practices.  The  opportunity 
exists  to  critically  study  these  elements  to  learn 
whether  they  are  valid  and  reliable  measures  of 
provider  performance. 
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Research  Questions 

Provider  (Institution  and  Clinician)  Uses  of 
Quality  Information 

1.  Very  little  of  the  published  literature  on  use  of 
information  in  QI  and  QA  meets  even  the  most 
liberal  rules  of  evidence.  A  key  question  is: 
What  are  the  efficacy  and  effectiveness  of  vari- 
ous models  for  QI  and  QA  efforts  when  scientif- 
ically and  objectively  measured? 

2.  What  is  the  diagnostic  accuracy  (sensitivity  and 
specificity)  of  various  quality  measures  as 
applied  by  institutions  in  their  quality  related 
efforts?  Measurement  using  an  objective  stand- 
ard is  a  criterion  for  diagnostic  maneuver  assess- 
ments and  is  lacking  in  assessment  of  the  meas- 
ures used  in  quality  information  generation. 

3.  What  are  the  relative  benefits  of  various  compo- 
nents of  both  QI  and  Q  A?  For  example,  the 
benefit  of  benchmarking  information  is  a  key  QI 
issue;  the  benefit  of  components  such  as  medical 
record  review  is  critical  for  typical  QA. 

4.  What  effect  do  comprehensive  models  of  quality 
information  use  (such  as  those  integrating  out- 
comes management  information  with  formal 
TQM)  for  QI  have  on  the  decisionmaking  and 
processes  of  institutions? 

5.  How  accurate  are  classifications  of  providers 
or  institutions  regarding  quality  of  care  based 
on  various  types  of  quality  information  (e.g., 
severity-adjusted  outcomes,  clinical  indicator 
rates,  meeting  standards  for  the  process  of  care)? 

Customer  (Purchaser  and  Patient)  Uses  of 
Quality  Information 

1.  A  number  of  questions  relate  to  "buy  right" 
programs:  How  have  they  been  implemented? 
What  have  been  the  quality  measures  used  to 
select  providers?  How  valid  and  reliable  are 
those  measures?  How  have  employees,  depen- 
dents, and  retirees  responded  to  incentives? 
How  has  the  health  care  community  responded 
to  the  incentives?  Have  the  programs  resulted 
in  a  change  in  the  delivery  of  health  care  in  the 
community?  How  do  we  know? 

2.  How  valid  and  reliable  are  the  quality  measures 
currently  being  used  by  purchasers  to  select  and 
monitor  health  care  providers  with  reference  to 
the  provider's  ultimate  performance?  How  do 


these  measures  differ  in  various  components 
of  care  delivery  (hospital,  physician,  nursing, 
mental  health,  highly  specialized  "centers  of 
excellence,"  personal  care,  and  so  forth)? 

3.  What  are  the  pros  and  cons  of  delegating  quality 
of  care  monitoring  to  external  bodies  (such  as 
government  agencies,  private  accrediting  bod- 
ies)? Is  there  a  taxonomy  of  assigned  responsi- 
bility for  health  care  quality  that  extends  from 
the  patient  to  the  payer,  to  the  health  care 
provider,  to  the  government? 

4.  What  are  the  legal,  political,  clinical,  and  ethical 
implications  of  including  quality  of  care  in 
contractual  language? 

5.  What  elements  of  customer  satisfaction  are  valid 
and  reliable  predictors  of  the  technical  quality 
of  care?  The  process  of  care? 

6.  What  are  the  elements  of  successful  and  unsuc- 
cessful "partnerships"  between  health  care 
customers  and  suppliers?  Do  partnerships 
improve  the  quality  of  care  delivered  in  a 
community?  How  do  we  know? 

7.  What  methods  of  information  brokering  are 
specifically  effective  in  educating  employees, 
dependents,  or  retirees?  Does  information 
dissemination  lead  to  changed  patient/consumer 
behavior  as  indicated  by  different  utilization 
patterns,  cost  changes,  and/or  quality  of  care 
measures? 

8.  How  do  patients  choose,  monitor,  and  terminate 
their  relationship  with  the  health  care  delivery 
system?  What  are  the  relative  roles  of  physi- 
cians, friends  and  family,  media,  publicly  avail- 
able data,  other  patients,  employers  and  insur- 
ers, advertising,  and  other  factors? 

9.  Is  there  a  taxonomy  of  patients  that  relates  to 
the  intensity  and  type  of  information  needed 
to  make  health  care  decisions  (that  is,  from 
no  information  to  every  possible  piece  of 
information)? 

10.  How  do  different  types  of  health  care  plans 
(fee-for-service  vs.  managed  indemnity  vs. 
different  types  of  HMOs)  affect  patient  decision- 
making and  delivery  of  health  care? 

11.  What  are  the  successful  models  of  turning  health 
care  data  into  health  care  information?  How 

do  we  know  they  are  successful?  How  is  the 
information  used? 
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12.  How  do/should  emerging  technologies  affect 
patient  decisionmaking  (interactive  video  discs, 
real-time  computer-assisted  systems  such  as 
APACHE)? 

13.  How  does  consumer/patient  participation  on 
health  care  boards,  committees,  and  other  orga- 
nizations affect  the  deliberations  and  decisions 
of  those  bodies  and  patient  care? 
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Presentation 


What  Is  Quality  Improvement:  A  Report  from  the  Field 

Eugene  C.  Nelson,  D.Sc.,  M.P.H. 


Introduction 

This  paper  concerns  quality  improvement  in 
the  field.  I  am  writing  from  the  vantage  point  of 
someone  who  is  working  in  the  field,  and  I  will 
explain  what  quality  improvement  (QI)  is  and 
where  research  is  needed. 

First,  many  of  the  best  and  brightest  health  care 
providers  are  working  very  hard  to  try  to  find  bet- 
ter ways  to  deliver  health  care  by  harnessing  QI 
theories  and  methods  and  putting  them  to  work  in 
hospitals,  clinics,  and  health  maintenance  organiza- 
tions (HMOs). 

Those  championing  QI  have  used  many  differ- 
ent terms — such  as  continuous  quality  improve- 
ment (CQI),  continual  improvement,  total  quality 
management,  and  organization-wide  improvement 
— to  describe  a  broad  theoretical  knowledge  base 
and  an  associated  set  of  practices,  skills,  tools,  and 
techniques.  Some  basics  of  QI  in  health  care  involve: 

•  Understanding  and  meeting  the  recognized  and 
unrecognized  needs  and  expectations  of  patients 
and  other  purchasers  of  health  care. 

•  Understanding  what  quality  means  from  the 
viewpoint  of  patients,  families,  and  other  pur- 
chasers of  health  care. 

•  Leading  health  care  organizations  seeking  to 
meet  needs  and  expectations  through  systematic 
efforts  to  improve  quality  and  value. 

•  Understanding  that  all  health  care  is  provided  in 
systems,  medical  procedures  are  interdependent 
with  the  delivery  system,  and  knowledge  of  these 
interdependencies  is  fundamental  to  improve- 
ment. 

•  Understanding  the  core  patient  care  delivery 
process  and  related  supporting  processes  in  the 
settings  where  care  is  delivered. 
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•  Evaluating  variation  in  patient  case-mix,  deliv- 
ery processes,  and  outcomes  as  a  guide  to  taking 
economical  action  on  processes  and  outcomes. 

•  Measuring  the  quality  and  value  of  health  care 
using  multiple  indicators  based  on  knowledge  of 
the  systems,  "customers,"  and  underlying  social 
need  for  health  and  health  care. 

•  Taking  action  for  improvement  (that  is,  planned 
improvement  trials)  that  simultaneously  build 
knowledge  on  how  to  take  action  effectively. 

•  Involving  the  energies  of  everyone  in  the  health 
care  organization  for  the  task  of  never-ending 
improvement. 

Although  these  ideas  are  not  new,  QI  explicitly 
grounded  in  these  theories  is  new  to  health  care, 
and  interest  is  mushrooming.  Numerous  articles  and 
books  are  educating  physicians  and  health  profes- 
sionals about  QI  ideas  and  practice.  Moreover,  the 
Joint  Commission  on  Accreditation  of  Healthcare 
Organizations  (JCAHO)  has  clearly  stated  its  inten- 
tion to  begin  emphasizing  continual  improvement 
of  patient  care  as  the  main  emphasis  of  health  care 
organizations. 

On  the  other  hand,  there  are  QI  skeptics,  both 
providers  and  researchers,  who  have  seen  countless 
fads  come  and  go.  They  have  seen  old  wine  put 
into  new  bottles,  and  they  ask:  What  is  QI?  Is 
there  anything  different  between  QI  and  quality 
assurance  except  the  swap  of  a  capital  I  for  a  capital 
A?  Does  QI  work?  How?  What  is  QI  research?  Is 
it  worth  doing? 

These  are  fair  questions,  and  they  demand  good 
answers.  I  propose  to  give  you  some  answers,  but 
rather  than  tell  you  what  I  personally  view  as  the 
answers,  I  will  take  a  different  approach  that  I  will 
call  QAE.  This  stands  for  "questions  and  answers 
from  experts." 

Questions  and  Answers  from  Experts 

In  preparing  this  paper  I  interviewed  two  expert 
QI  practitioners  who  have  dedicated  the  past  5 
years  of  their  lives  to  work  on  quality.  One  is  Paul 
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Batalden,  a  physician  who  is  the  senior  quality  offi- 
cer of  Hospital  Corporation  of  America  (HCA). 
The  other  is  Chip  Caldwell,  president  and  chief 
executive  officer  (CEO)  of  West  Paces  Ferry 
Hospital.  I  asked  each  of  them  four  questions: 

1.  What  is  the  essence  of  QI? 

2.  What  is  your  QI  work? 

3.  What  have  you  accomplished? 

4.  How  could  QI  research  be  put  to  work  for  you? 

What  is  the  essence  of  QI? 

Batalden:  The  essence  of  QI  work  is  that  it  is  done 
from  a  different  knowledge  base  than  we  have  used 
to  make  improvement  before.  An  example  is  small- 
pox. We  knew  the  basic  science  of  smallpox — the 
virology,  the  immunology,  and  the  pathophysiology 
— decades  before  we  were  able  to  effectively  apply 
that  knowledge.  In  fact,  we  were  within  a  hair's 
breadth  of  eliminating  smallpox  in  Bangladesh  when 
the  Muslim  festival  of  Eid  occurred.  The  entire 
population  packed  up  their  belongings  and  visited 
their  birth  homes,  much  to  the  surprise  and  chagrin 
of  the  public  health  workers.  They  lacked  knowl- 
edge of  the  system  of  disease  and  the  sources  of 
variation.  This  limited  the  effectiveness  of  their 
work.  As  a  result,  smallpox  persisted  for  3  more 
years  in  Bangladesh. 

The  knowledge  that  is  the  essence  of  QI  and 
CQI  includes  the  traditional  discipline-specific 
knowledge  of  health  care  complemented  by  knowl- 
edge of  the  system;  knowledge  of  variation  and  the 
way  it  manifests  itself  in  that  system;  knowledge  of 
psychology,  particularly  the  intrinsic  motivation  that 
is  present  within  the  system  and  can  be  made  avail- 
able for  improvement  with  wise  management;  and 
finally,  a  theory  or  approach  to  taking  action  that 
simultaneously  builds  further  knowledge  of  the 
system  and  its  operations. 

Thus,  the  essence  of  CQI  has  to  do  with  a  differ- 
ent underlying  knowledge  system  than  the  knowl- 
edge system  that  supports  our  traditional  efforts  at 
improvement. 

Caldwell:  People  have  tried  to  make  QI  a  mystery. 
The  first  time  I  saw  the  Deming  cycle,  it  struck  me  as 
the  purest  integration  of  marketing  and  operations  I 
had  ever  seen.  I  remember  the  first  time  I  talked  to 
Dr.  Tommie  Frist,  the  CEO  of  HCA,  about  the 


Deming  cycle  and  QI.  He  said:  "How  can  anybody 
be  against  this?  It's  just  good  management." 

I  see  QI  as  a  framework  for  understanding  the 
need  to  provide  health  care  and  to  interpret  and 
reinterpret  customer  needs  as  they  change  over  time. 
This  enables  you  to  make  continual  improvement  in 
processes  and  services  to  meet  these  changing  needs. 
In  short,  QI  is  nothing  more  than  a  framework  to 
achieve  what  you  are  in  business  to  do. 

What  is  your  QI  work? 

Batalden:  I  see  myself  as  a  student.  My  job  is  to 
study  the  basic  knowledge  of  improvement  and  to 
work  together  with  a  wide  range  of  others  to  test 
applications  of  that  knowledge  in  the  daily  work  of 
health  care  and  hospitals.  As  such,  I  find  myself 
continually  walking  between  the  world  of  theory 
and  the  world  of  practice  and  involved  in  a  never- 
ending  effort  to  simplify  and  clean  up  the  ways  in 
which  I  communicate  about  that  journey  between 
theory  and  practice. 

Caldwell:  First,  I  use  QI  tools  to  learn  which 
businesses  the  hospital  should  be  in.  This  means 
synthesizing  all  the  myriad  customers  and  their 
sometimes  conflicting  needs.  Second,  once  we  have 
created  a  new  business  or  service — such  as  the 
Diabetes  Center  or  the  Sports  Medicine  Center — I 
supervise  it  using  QI  tools.  I  have  the  managers  of 
the  services  use  the  same  QI  tools  so  they  can  tell 
me  how  well  they  are  doing. 

Third,  I  help  ensure  that  we  have  an  empowered 
work  force.  The  challenge  is  to  create  and  monitor 
an  environment  where  QI  tools  are  used  by  every- 
one. To  do  this  we  have  worked  to  create  a  shared 
vision,  and  we  have  developed  a  set  of  42  inte- 
grated, hospital-wide  measures  that  link  with  400 
measures  at  the  department  level.  These  measures 
give  us  a  compass  to  see  where  we  are  and  if  we 
are  heading  in  the  right  direction.  With  regard  to 
empowerment,  it  is  irresponsible  for  management 
to  withhold  the  power  from  individuals  and  teams 
to  make  improvements.  If  management  wants  to 
empower  the  work  force,  then  we  must  educate 
them  to  recognize  whether  a  proposed  change  is 
good  or  bad  for  the  whole  hospital.  That  is,  employ- 
ees need  criteria  they  can  use  to  tell  if  a  change  is 
really  an  improvement  for  the  hospital  as  a  whole. 

Fourth,  I  am  working  to  build  a  care  delivery 
system  that  self-improves.  We  are  working  with 
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physicians  and  other  clinicians  to  make  repeated 
improvements  in  areas  like  maternity  care  and 
cardiac  care  and  for  chronic  problems  such  as 
diabetes.  For  example,  we  are  trying  to  develop  a 
system  for  providing  diabetes  care  that  brings 
annual  health  care  costs  down  while  simultaneously 
improving  outcomes. 

What  have  you  accomplished? 

Batalden:  Not  very  much,  in  relation  to  the  task.  I 
have  had  the  privilege  of  identifying  and  working 
with  a  small  number  of  colleagues  in  Nashville  and 
a  larger  number  of  dedicated  health  leaders  in 
hospitals  and  health  systems  across  the  country. 
Whatever  I  have  accomplished  has  been  in  conjunc- 
tion with  and  through  them.  Here  are  some  of  our 
achievements: 

1.  We  have  developed  a  family  of  measures  that 
can  be  used  by  health  care  leaders  to  measure 
the  judgments  of  quality  made  by  patients  and 
their  families,  communities,  employees,  and 
payers.  Leaders  can  use  these  customer-based 
measures  of  quality  to  improve  services. 

2.  We  have  developed  an  approach  for  health  care 
leaders  to  introduce  this  new  knowledge  (that 
is,  continual  improvement)  and  an  associated 
curriculum  of  learning  opportunities  to  support 
those  leaders. 

3.  We  have  shared  our  work  broadly  with  graduate 
programs  in  health  administration,  medicine, 
and  nursing  at  places  such  as  Duke  University 
and  the  University  of  Toronto. 

4.  We  have  helped  the  JCAHO  develop  an  under- 
standing of  continual  improvement. 

5.  We  have  worked  with  nearly  100  hospitals  and 
health  care  systems  inside  and  outside  of  HCA 
and  the  Henry  Ford  Health  System  to  help  them 
make  many  improvements. 

Caldwell:  These  are  a  few  of  my  hospital's  accom- 
plishments. 

1.  We  have  cut  cesarean  section  rates  from  22  per- 
cent to  under  12  percent  in  3  years. 

2.  We  have  increased  our  efficiency  in  treating 
chest  pain  (that  is,  admitting  patients  with  actual 
myocardial  infarction  (MI)  as  opposed  to 
patients  with  non-MI  chest  pain)  from  28  per- 
cent to  68  percent.  This  saved  one  of  our  cus- 
tomers, Prucare,  $330,000  in  1  year. 


3.  We  have  reduced  the  percentage  of  non-regis- 
tered nursing  duties  that  nurses  perform  from  40 
percent  to  24  percent  and  have  saved  $250,000 
annually. 

4.  We  have  cut  turnaround  time  in  the  operating 
room  by  25  percent,  thereby  cutting  the  amount 
of  time  that  patients  have  to  wait  in  the  holding 
area  from  23  to  6  minutes. 

5.  We  have  created  the  expectation  among  our 
medical  staff  that  any  new  service  we  choose  to 
offer  will  be  run  very  well  and  will  be  top  quality. 
For  those  interested  in  dollars,  we  have  increased 
annual  operating  margins  from  $9  million  to 
$20  million. 

How  could  Ql  research  be  put  to  work 
for  you? 

Batalden:  Helpful  research  requires  careful  under- 
standing of  the  context  to  which  it  will  be  applied. 
Therefore,  QI  research  will  be  most  helpful  if  it  is 
undertaken  with  an  understanding  of  theory  and 
the  realities  facing  leaders  who  are  trying  to  apply 
that  theory. 

Given  those  requirements  and  caveats,  research 
in  these  areas  could  be  useful: 

1.  New  knowledge  about  organizational  change. 

2.  Behavior  of  health  care  processes  and  systems. 

3.  Underlying  needs  and  drivers  of  quality  assess- 
ments by  patients  and  their  families,  communi- 
ties, providers,  and  payers. 

4.  More  effective  means  of  teaching  scientific  and 
statistical  thinking  to  health  workers. 

5.  More  effective  recognition  mechanisms  for  the 
discernment  of  barriers  to  joy  and  pride  in  work. 

6.  Balanced  measures  of  provider  effectiveness  in 
improving  health  risk  and  health  status. 

7.  Ways  of  reducing  the  costs  of  applying  CQI 
knowledge  into  the  daily  work  of  ordinary 
nurses,  doctors,  and  hospitals  and  other  health 
care  settings. 

8.  Better  information  on  ways  to  educate  health 
workers  about  measurement — for  confirming  that 
we  are  doing  and  improving  as  we  intend,  and  for 
learning  how  we  can  make  even  more  improve- 
ments in  our  processes  and  systems  of  work. 

Caldwell:  I  will  answer  this  question  with  my  wish 
list.  I  wish  we  had: 
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1.  More  knowledge  on  how  to  make  quality 
deployment  throughout  my  hospital  easier. 

2.  An  automated  medical  information  system  to 
link  all  the  needed  data  together. 

3.  More  "time"  data,  (that  is,  showing  variability  in 
how  long  it  takes  to  do  things)  to  complement 
our  financial  data  (that  is,  showing  how  much 
things  cost). 

4.  Methods  for  working  with  our  suppliers  to 
reduce  the  total  costs  of  services  we  provide;  for 
example,  research  on  how  to  reduce  supply  costs 
in  our  most  labor  intensive  areas. 

5.  Knowledge  about  better  employee  compensa- 
tion systems. 

6.  Better  methods  for  teaching  employees  how  to 
use  QI  tools  to  improve  their  daily  work. 

7.  Benchmarking  systems  to  help  us  learn  how  to 
provide  services  more  effectively. 

8.  Data  systems  that  employers  could  use  to  accu- 
rately identify  providers  who  deliver  the  best  and 
most  efficient  care  for  a  population  of  patients. 

9.  Research  results  on  care  delivery  models  for  the 
top  five  diagnoses  in  the  Nation  that  show  how  to 
make  health  status  and  efficiency  improvements. 

10.  Research  on  how  to  develop  community  models 
for  health  care  delivery  that  would  enable  all 
providers  to  work  together  to  improve  quality 
and  cut  costs  (for  example,  20  hospitals  in 
Atlanta  are  working  together  to  find  safe  ways 
to  cut  the  rate  of  cesarean  sections  by  eliminat- 
ing unnecessary  ones). 

Conclusion 

I  have  presented  comments  from  two  of  the  best 
thinkers  and  practitioners  of  QI  in  health  care,  both 
of  whom  have  committed  the  past  5  years  of  their 
professional  lives  to  learning  about  and  practicing 
a  new  way  to  continually  improve. 

In  his  answer  to  the  first  question,  "What  is  the 
essence  of  QI?"  it  is  apparent  that  Batalden  sees  QI 
as  a  fundamentally  new  system  of  knowledge  that 
complements  existing  knowledge.  Caldwell  views  it 
in  more  concrete  terms  as  just  a  framework,  albeit 
the  most  powerful  one  he  has  ever  seen,  for  achiev- 
ing his  hospital's  organizational  objectives.  When 
asked  "What  is  QI  work?"  both  experts  tell  us  that 
QI  is  not  just  one  part  of  what  they  do  everyday,  it 


is  their  daily  work.  Batalden  tells  how  his  continual 
improvement  work  is  tied  up  with  continual  learn- 
ing and  devising  sure  ways  to  connect  theory  and 
practice.  His  job  is  to  teach  health  care  leaders. 
To  paraphrase  the  utilitarian  philosopher  James 
Dewey,  the  goal  is  not  to  find  ways  to  make  theory 
simple  but  to  make  practice  wise. 

Caldwell's  QI  work  is  fully  integrated  into  his 
daily  work  as  head  of  a  hospital.  This  involves 
strategic  planning,  supervising,  measuring,  empow- 
ering all  employees,  and  searching  for  clinical  care 
innovations  to  meet  community  needs. 

On  the  third  question,  "What  have  you 
accomplished?"  it  is  clear  that  both  men  have 
accomplished  a  great  deal,  but  there  is  an  enor- 
mous amount  of  further  work  that  must  be  done. 
This  is  bad  news  for  people  who  are  looking  for 
quick  fixes.  Health  care  systems  are  big  and  compli- 
cated. Making  substantial  improvements  in  large 
health  care  organizations  with  respect  to  outcomes, 
efficiency,  access,  costs,  and  satisfaction  will  take 
years  not  months.  Ford  Motor  Company  has  been 
working  on  quality  for  nearly  10  years.  Its  execu- 
tives think  they  are  about  20  percent  along  the  way 
of  fully  using  a  QI  process,  but  Deming  puts  it  at 
less  than  10  percent. 

Besides  taking  time,  if  health  care  providers  are 
to  learn  to  accelerate  their  ability  to  improve  quality 
and  value,  they  will  need  to  cooperate  with  each 
other  in  new  ways,  and  they  will  need  reinforcement 
from  business  leaders,  policymakers,  regulators, 
and  researchers. 

As  for  the  fourth  question,  "How  could  QI 
research  be  put  to  work?"  it  is  clear  that  in  theory 
quality  research  could  be  extremely  valuable  for 
improved  quality,  decreased  costs,  and  greater  over- 
all value  in  health  care.  But  to  actually  help  health 
care  providers,  who  are  key  beneficiaries  of  quality 
research,  it  must  be  grounded  in  a  firm  understand- 
ing of  QI  theory  and  practice  and  the  system  of  care 
delivery  as  it  actually  exists,  with  its  constraints  and 
accelerators,  its  barriers  and  supports. 
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Introduction 

To  quote  Heather  Palmer  (in  this  volume), 
"Quality  is  defined  as  meeting  the  needs  and  expec- 
tations of  customers."  There  are  many  customers, 
of  course — some  internal,  some  external — but  the 
ultimate  customer  is  the  patient;  if  this  enormous 
health  care  enterprise  is  about  anything,  it  is  about 
meeting  the  needs  and  expectations  of  patients. 

My  purpose  for  the  conference  and  for  this 
paper  is  to  speak  to  you  as  a  patient — a  far  more 
frequent  patient  than  I  might  have  wished — and 
from  that  vantage  point  to  tell  you  how  we  are 
doing  and  what  we  need  to  do. 

We  are  doing  poorly.  The  health  care  system 
fails  the  Palmer  test:  it  does  not  meet  the  needs  and 
expectations  of  patients;  it  does  not  create  satisfied 
customers,  which  Peter  Drucker  calls  the  only  valid 
business  purpose  (Moloney  and  Paul,  1991).  The 
health  care  system  fails  to  do  so  because  under  the 
prevailing  system  of  doctor-patient  relations,  patient 
preferences  play  only  a  minor  role;  the  demand  for 
medical  care  is  largely  physician-induced.  True, 
patients  initiate  most  medical  encounters,  but  they 
have  little  to  say  about  what  happens  thereafter; 
doctors,  not  patients,  decide  whether  patients  will 
be  hospitalized  and  whether  they  will  undergo 
surgery,  chemotherapy,  or  some  other  treatment  or 
procedure.  Physicians  make  these  decisions  with  the 
best  of  intentions;  they  genuinely  seek  to  meet 
the  needs  of  their  patients.  But  physicians — not 
patients — define  those  needs.  Physicians  usually  do 
so,  moreover,  without  asking  patients  what  they 
want  or  think  they  need  and  without  otherwise 
taking  patients'  preferences  and  values  into  account. 

Physicians  make  these  decisions  unilaterally, 
not  because  they  crave  power  but  because,  until 
recently,  everybody  took  it  for  granted  that  "doctor 
knows  best."  That  assumption,  in  turn,  rested  on 
four  other  assumptions,  namely: 
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1.  Physicians  know  the  scientifically  correct  way  to 
practice  medicine. 

2.  They  know  the  values  and  preferences  of  their 
patients. 

3.  Physicians  are  disinterested  and  objective  scien- 
tists without  preferences  or  biases  of  their  own. 

4.  They  therefore  routinely  choose  for  their 
patients  what  those  patients  would  choose  for 
themselves  if  they  had  the  same  knowledge. 

None  of  these  assumptions  can  withstand 
scrutiny.  As  Jack  Wennberg  (1984, 1987, 1988; 
Wennberg,  Freeman,  and  Culp,  1987),  Robert 
Brook  (1991;  Brook  and  McGlynn,  1991;  Winslow, 
Solomon,  Chassin,  and  others,  1988),  David  Eddy 
(1982, 1984, 1990a,  1990b;  Eddy  and  Billings,  1988), 
and  others  have  shown,  physicians  frequently  do  not 
know  the  scientifically  correct  way  to  practice  medi- 
cine— not  because  of  personal  ignorance  or  indiffer- 
ence, but  because  nobody  knows;  medical  science 
has  enjoyed  spectacular  success  in  developing  new 
diagnostic  tests  and  treatments  but  has  failed  to 
evaluate  the  usefulness  of  those  tests  and  the  out- 
comes of  those  treatments.  That,  after  all,  is  why  the 
Agency  for  Health  Care  Policy  and  Research  was 
brought  into  being.  Like  other  human  beings,  more- 
over, physicians  have  biases  and  preferences  of  their 
own;  they  usually  do  not  know,  and  indeed,  often 
misjudge  their  patients'  values  and  preferences. 

These  misjudgments  are  rooted  in  an  enormous 
gulf  between  the  way  physicians  think  about  disease 
and  the  way  physicians  experience  it — a  gulf  physi- 
cians often  discover  when  they  become  seriously 
ill  themselves.  "I  practiced  medicine  for  50  years 
before  I  became  a  patient,"  Edward  E.  Rosenbaum, 
M.D.,  (1988)  former  chief  of  rheumatology  at 
Oregon  Health  Sciences  University,  wrote  in  the 
book  on  which  the  film,  The  Doctor,  was  based.  "It 
wasn't  until  then  that  I  learned  that  the  physician 
and  patient  are  not  on  the  same  track.  The  view  is 
entirely  different  when  you  are  standing  at  the  side 
of  the  bed  from  when  you  are  lying  in  it." 
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Perceptions  of  Illness  and  Disease 

To  the  seriously  ill  patient,  disease  is  an  assault 
on  identity  and  sense  of  self,  an  assault  involving 
disorientation  and  loss  of  meaning,  erosion  of 
autonomy  and  control,  fear,  and  dependency.  Sick- 
ness also  is  a  transforming  experience.  The  transfor- 
mation is  most  evident  when  sickness  is  chronic  or 
progressive  or  leaves  us  with  a  permanent  disability. 
But  even  if  we  "recover,"  we  do  not  return  to  our 
pre-sickness  state;  we  must  adjust  to  a  new  way  of 
seeing  and  relating  to  the  world.  To  the  patient, 
disease  always  is  an  existential  process. 

To  the  physician,  on  the  other  hand,  disease  is  a 
pathophysiological  process — one  that  is  abstracted 
from  the  patient  and  his  or  her  suffering.  To  a 
neurologist,  for  example,  multiple  sclerosis  (MS)  is 
a  "slowly  progressive  central  nervous  system  disease 
characterized  by  disseminated  patches  of  demyelina- 
tion  in  the  brain  and  spinal  cord,  resulting  in 
multiple  and  varied  neurologic  symptoms  and  signs, 
usually  with  remissions  and  exacerbations." 

That  is  not  how  MS  patients  experience  the  dis- 
ease. "What  I  experience  every  day  is  not  the 
demyelination  in  the  white  matter  of  the  nervous 
system,"  philosopher  Kay  Toombs  (1992;  see  also 
Toombs,  1989)  explains,  but  the  "ongoing  and  seem- 
ingly relentless  diminishment  of  physical  abilities 
which  is  gradually  but  surely  eroding  my  indepen- 
dence. For  me,  multiple  sclerosis  is  the  constant 
effort  to  overcome  my  body's  resistance  so  that  I  can 
carry  out  the  most  mundane  activities;  it  is  the  frus- 
tration of  not  being  able  to  do  the  simplest  of  things. 
It  is  the  anguished  uncertainty  of  a  perilous  future." 

I  know  what  Dr.  Toombs  means.  For  me,  a  brain 
stem  infarct — in  laymen's  terms,  a  stroke — was  not 
"an  intra-  or  extracranial  interruption  in  arterial 
blood  flow,"  as  the  Merck  Manual  (Berkow  and 
Fletcher,  1987)  describes  it.  It  was  the  infantilizing 
inability  to  stand  without  collapsing  to  my  left  and 
the  unimaginable  terror  that  what  already  had 
happened  to  my  hospital  roommates  would  soon 
happen  to  me — that  I  would  lose  the  ability  to 
speak,  think,  write,  or  feed  myself. 

For  patients,  disease  never  is  just  a  pathophysio- 
logic process.  "What  happens  when  my  body  breaks 
down  happens  not  just  to  that  body  but  also  to  my 
life,  which  is  lived  in  that  body,"  explains  Arthur 
Frank  (1991),  a  medical  sociologist  who  had  a 
massive  coronary  followed  by  testicular  cancer. 
"When  the  body  breaks  down,  so  does  the  life." 
Fixing  the  body  "doesn't  always  put  the  life  back 


together  again,"  Frank  adds,  because  serious  illness 
"leaves  no  aspect  of  life  untouched . . .  your  relation- 
ships, your  work,  your  sense  of  who  you  are  and 
who  you  might  become,  your  sense  of  what  life 
is  and  ought  to  be — these  all  change,  and  the 
change  is  terrifying." 

Terrifying,  as  I  can  attest,  because  sickness 
plunges  us  into  an  abyss  of  meaninglessness;  we 
know  what  our  symptoms  are  but  not  what  they 
mean.  Sickness  also  is  disorienting,  for  it  shatters 
the  web  of  assumptions  on  which  our  lives  are 
based.  We  take  it  for  granted,  for  example,  that 
our  arms,  legs,  fingers,  feet,  and  other  organs  will 
respond  to  our  usually  unconscious  commands.  It  is 
only  when  that  does  not  happen — when  we  cannot 
stand  without  falling,  as  happened  to  me  after  my 
stroke,  or  when  bringing  a  cup  of  coffee  to  our  lips 
without  spilling  it  requires  all  the  concentration  we 
can  muster — that  we  discover  the  degree  to  which 
our  sense  of  self  had  depended  on  those  assump- 
tions and  how  disoriented  we  become  when  our 
body  turns  into  an  enemy  rather  than  an  ally. 

No  wonder  patients  and  doctors  have  such 
difficulty  communicating  with  one  another.  "In 
discussing  my  illness  with  physicians,"  Kay  Toombs 
(1987)  reports,  "it  has  often  seemed  to  me  that  we 
have  been  talking  at  cross-purposes,  discussing 
different  things,  never  quite  reaching  one  another." 
For  the  most  part,  Dr.  Toombs  adds,  "this  inability 
to  communicate  does  not  result  from  inattentive- 
ness  or  insensitivity  but  from  a  fundamental 
disagreement  about  the  nature  of  illness.  Rather 
than  representing  a  shared  reality  between  us, 
illness  represents  two  quite  different  realities,  the 
meaning  of  one  being  significantly. . .  different 
from  the  meaning  of  the  other." 

Improving  the  Quality  of  Health  Care 

We  cannot  improve  the  quality  of  medical  care 
unless  both  realities  are  incorporated  into  clinical 
practice.  Because  doctors  and  patients  understand 
disease  so  differently,  they  have  radically  different 
expectations  of  medical  care  and,  therefore, 
radically  different  ways  of  assessing  its  quality.  The 
assumption  that  disease  is  physiological  leads 
doctors  to  set  physiological  goals  and  to  measure 
the  quality  of  care  through  "intermediate  outcomes" 
— changes  in  physiological  measures  such  as  blood 
pressure,  blood  sugar  levels,  tumor  size,  or  patency 
of  blood  vessels. 
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These  are  important  outcomes,  but  they  are 
incomplete,  at  best,  and  often  tell  the  doctor  little 
about  how  patients  feel  and  function,  the  outcomes 
that  interest  patients  most.  In  a  host  of  chronic  ill- 
nesses— conditions  such  as  ulcers,  angina,  arrhyth- 
mia, diabetes,  spinal  stenosis  and  disk  herniation, 
and  benign  enlargement  of  the  prostate,  among  oth- 
ers— there  is  remarkably  little  correlation  between 
the  severity  of  the  organic  disorder  and  the  amount 
of  pain  or  disability  patients  experience;  the  same 
pathophysiology  is  associated  with  widely  different 
symptoms  in  different  patients.  By  concentrating  on 
physiological  measures,  a  rheumatologist  observes, 
physicians  "may  all  too  easily  spend  years  writing 
'doing  well'  in  the  notes  of  a  patient  who  has 
become  progressively  more  crippled  before  their 
eyes"  (Deyo,  1991;  see  also  Smith,  1983). 

That  is  not  how  patients  define  "doing  well."  By 
and  large,  patients  judge  the  effectiveness  of  a  drug 
or  procedure  by  its  impact  on  their  lives.  They  want 
to  know  whether  the  treatment  will  relieve  pain  or 
alleviate  anxiety;  whether  it  will  preserve  or  restore 
their  ability  to  see,  hear,  walk,  climb  stairs,  or  dress 
and  feed  themselves,  and  whether  it  will  change 
their  appearance,  sexuality,  or  mood. 

In  short,  patients  judge  the  quality  of  care  by  its 
impact  on  the  quality  of  their  lives — an  outcome 
physicians  and  hospital  administrators  rarely  meas- 
ure. Clinicians  collect  information  about  the  func- 
tioning of  almost  every  bodily  organ  but  not  about 
the  functioning  of  patients  as  a  whole.  "We  appraise 
the  palliative  treatment  of  patients  with  cancer  by 
measuring . . .  tumor  size  and  other  paraclinical 
indexes,"  Alvan  R.  Feinstein  (1983)  points  out. 
"We  do  not  regularly  measure  whether  the  patient 
is  comfortable  or  miserable,  functional  or  bed- 
ridden, vegetating  or  vibrant.  We  compare  medical 
and  surgical  treatment  for  coronary  heart  disease 
by  measuring . . .  patency  of  vessels,  changes  in 
electrocardiograms,  and  treadmill  exercise  tests. 
We  do  not  regularly  perform  scientifically  credible 
measurements  of  whether  the  angina  pectoris  is  still 
severe,  whether  the  patient  was  truly  made  able  to 
return  to  work,  and  whether  the  quality  of  life  has 
otherwise  improved  for  the  patient  and  his  family." 

These  failures  are  due  not  to  insensitivity  or  lack 
of  concern  but  to  the  prevailing  mind-set  about  the 
kind  of  information  physicians  should  seek  and 
study.  By  and  large,  physicians  are  taught  that  what 
cannot  be  readily  quantified  is  unimportant;  infor- 
mation about  what  patients  think  and  how  they  feel 


or  function  has  been  considered  "soft"  or  "subjec- 
tive" and  therefore  less  worthy  of  scientific  attention 
than  "hard"  or  "objective"  data,  such  as  the  images, 
numbers,  and  tracings  produced  by  diagnostic  test- 
ing equipment.  By  making  this  invidious  distinction, 
Feinstein  (1987)  notes,  physicians  ignore  "all  the 
distinctively  human  reactions — love,  hate,  joy, 
sorrow,  distress,  gratification — that  differentiate 
people  from  animals  or  molecules." 

Ignoring  or  slighting  the  human  dimensions  of 
medical  care  is  scientifically  untenable;  it  grows  out 
of  an  obsolete  view  of  the  nature  of  science.  It  also 
is  unnecessary.  Physicians  and  hospital  administra- 
tors now  have  at  their  disposal  a  number  of  easily 
used,  as  well  as  valid  and  reliable,  instruments  for 
assessing  patients'  symptoms,  functional  status,  and 
sense  of  well  being.  As  Sheldon  Greenfield  (in  this 
volume)  points  out,  some  of  these  measures  are 
actually  "harder" — that  is,  more  valid  and  reliable 
— than  many  of  the  physiologic  measures  in  common 
use.  The  availability  of  these  measures  makes  it 
scientifically  as  well  as  ethically  intolerable  for 
providers  to  fail  to  be  as  concerned  with  their 
patients'  functional  status  and  well-being  as  they 
are  with  their  patients'  physiologic  status. 

Physicians  and  hospital  administrators  no  longer 
have  a  choice.  Consumerism  is  rampant;  Business 
Week  (King  customer,  1990)  has  called  the  1990s 
"the  decade  of  the  customer."  As  Thomas  Moloney 
and  Barbara  Paul  (1991)  point  out,  members  of 
the  so-called  "baby  boom"  generation  want  far 
more  information  than  their  elders  about  the  ser- 
vices they  buy,  including  medical  care.  Indeed,  the 
Picker-Commonwealth  survey  that  Paul  Cleary  and 
colleagues  (1991)  describe  found  that  Americans 
are  eager  to  use  survey  data  to  help  them  decide 
where  to  seek  medical  and  hospital  care. 

Thus,  periodic  surveys  of  patient  attitudes  and 
experiences  must  be  incorporated  into  any  quality 
improvement  effort;  and  assessments  of  patients' 
functional  status  and  well-being  must  become  a 
routine  part  of  medical  and  hospital  care.  But  sur- 
veys and  assessments  are  not  enough.  Patients  are 
demanding  more  than  information;  they  want  to 
play  an  active  role  in  decisions  about  their  own  care. 

That  demand  must  be  met;  collaboration 
between  patient  and  physician  is  a  scientific,  ethical, 
and  economic  imperative.  An  ethical  imperative 
because  there  is  more  than  one  medically  appropri- 
ate treatment  for  almost  every  illness  or  condition, 
and  because  each  treatment  poses  a  different  set  of 
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risks  and  benefits  whose  probabilities  often  are  not 
known.  Because  patients  differ  widely  in  the  degree 
to  which  they  are  bothered  by  a  particular  symptom, 
moreover,  they  attach  widely  different  values  to  its 
relief.  Patients  also  differ  widely  in  their  willingness 
to  assume  the  risks  of  treatment.  Much  of  the  time, 
therefore,  there  is  no  medically  "right"  or  "wrong" 
decision;  what  is  "appropriate"  for  one  patient  may 
be  wholly  inappropriate  for  another.  Choice  of 
treatment  must  involve  collaboration  between 
doctors  and  patients. 

Collaboration  between  doctor  and  patient 
should  not  be  limited  to  the  choice  of  treatment;  it 
must  start  when  the  patient  first  seeks  medical  help 
and  continue  throughout  the  clinical  encounter.  If 
quality  is  defined  as  meeting  the  needs  of  the  cus- 
tomer, the  physician  must  find  out  what  those  needs 
are.  Something  has  happened,  after  all,  to  make  that 
patient  seek  medical  help.  Without  knowing  what 
that  something  is — what  caused  the  patient  to  come 
and  see  the  doctor  (and  patients  are  not  always 
aware  of  the  real  reason  themselves) — the  physician 
may  identify  and  treat  the  wrong  problem. 

The  risk  of  treating  the  wrong  problem  is 
increased  by  the  staccato,  close-ended  style  of  ques- 
tioning most  physicians  favor,  a  style  that  makes  it 
hard  for  patients  to  reveal  what  is  on  their  minds.  In 
analyzing  doctor-patient  interactions,  Beckman  and 
Frankel  (1984)  found  that,  on  average,  physicians 
interrupted  patients  18  seconds  after  the  patient 
began  to  speak;  patients  were  able  to  complete  their 
opening  statements  in  only  one  visit  in  four.  Time 
is  not  the  problem;  when  patients  were  permitted 
to  complete  their  statements,  they  took  only  60 
seconds  longer. 

Indeed,  interrupting  the  patient  may  lengthen 
the  visit  rather  than  shorten  it.  Witness  the  familiar 
situation  in  which,  just  as  the  physician  thinks  the 
office  visit  is  over,  the  patient  asks,  "Oh,  by  the  way 
doctor,  I  have  been  having  this  heavy  feeling  in  my 
chest.  Do  you  think  it  might  be  important?"  Most 
of  the  time  patients  are  not  being  perverse;  rather, 
they  have  not  been  able  to  get  the  question  out 
when  the  doctor  takes  their  history,  or  the  doctor 
may  not  put  them  at  sufficient  ease  to  enable  them 
to  ask  an  embarrassing  question  or  one  they  are 
afraid  to  ask  because  its  answer  may  be  frightening. 

The  only  way  patients  can  tell  physicians  why 
they  are  seeking  medical  help  is  through  genuine 
dialog;  and  genuine  dialog  cannot  occur  unless 
physicians  establish  comfortable  relationships  with 


patients.  "To  be  humane  and  empathic  is  not  merely 
a  prescription  for  compassion,"  George  Engel 
(1988)  explains.  "It  is  a  prerequisite  for  scientific 
work  in  the  clinical  realm." 

Paying  "proper  attention  to  the  full  needs  and 
feelings  of  patients"  (the  formulation  is  that  of 
Oliver  Sacks,  1987)  is  not  an  impossible  task. 
Thomas  Delbanco  (1992)  of  Beth  Israel  Hospital 
describes  one  way  of  doing  that.  In  their  encounters 
with  patients,  Delbanco  suggests,  doctors  should 
conduct  a  "patient  review"  comparable  to  the 
organ-specific  review  of  physiologic  symptoms  that 
is  part  of  every  physician's  routine.  Delbanco 
describes  seven  dimensions  of  care  that  need  to  be 
reviewed  and  suggests  questions  the  physician  might 
ask  in  each  dimension.  "Incorporating  the  patient's 
review  into  my  interactions  with  patients  encour- 
ages me  to  address  the  nontechnical  aspects  of  care 
in  a  systematic  way,"  he  writes. 

Conclusion 

To  sum  up,  quality  improvement  involves  more 
than  measurement;  it  requires  a  broadening  of  the 
goals  of  clinical  practice  and  a  transformation  of  the 
patient-doctor  relationship.  The  best  physicians  have 
always  understood  this.  As  Francis  Weld  Peabody 
told  his  Harvard  Medical  School  students  65  years 
ago,  "One  of  the  essential  qualities  of  the  clinician  is 
interest  in  humanity,  for  the  secret  of  the  care  of  the 
patient  is  in  caring  for  the  patient."  It  may  not  have 
been  entirely  coincidental  that  when  he  delivered  his 
classic  lecture  on  "The  Care  of  the  Patient,"  Dr. 
Peabody  was  himself  a  patient,  under  treatment  for 
the  cancer  that  killed  him  later  that  year. 
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Summary  Presentation 

Closing  Quality  Improvement/Quality  Assurance  Information  Gaps: 
AHCPR's  Role 

John  Ball,  M.D.,  J.D. 


Introduction 

For  me  this  was  a  learning  conference,  and  it  will 
be  difficult  for  me  to  say  something  new.  Instead,  I 
will  try  to  summarize  and  briefly  discuss  the  themes 
that  came  out  during  the  presentations. 

First,  you  should  know  that  I  am  speaking  from 
the  perspective  of  a  physician-clinician.  David 
Sundwall  and  I  planned  our  remarks  to  be  somewhat 
complementary.  David  will  discuss  the  physician  in 
the  hospital  and  management  role,  and  I  will  focus 
on  the  physician  in  a  clinical  role. 

I  also  bring  a  personal  history  to  my  remarks. 
About  20  years  ago,  I  was  chief  of  the  medical  audit 
branch  of  the  division  of  peer  review  in  the  Bureau 
of  Quality  of  Assurance.  My  responsibility  was  to 
design  and  implement  the  policy  that  the  profes- 
sional standards  review  organizations  would  use 
in  quality  assurance  (QA).  I  also  had  the  ancillary 
task  of  helping  to  determine  which  medical  proce- 
dures Medicare  would  cover.  It  is  interesting  to 
see  where  things  are  now.  I  worked  with  James 
Roberts,  who  is  with  the  Joint  Commission  on 
Accreditation  of  Healthcare  Organizations;  Dale 
Schumacher,  president  of  the  Commission  on 
Professional  and  Hospital  Activities;  William 
Jessee,  who  is  working  with  QA  at  Humana;  Mark 
Chassin,  who  is  the  commissioner  designate  for 
health  for  the  State  of  New  York;  and  finally,  Helen 
Smits,  who  was  director  of  the  health  standards  and 
quality  bureau,  Health  Care  Financing  Administra- 
tion. We  were  a  group  of  people  who  believed  QA 
had  a  lofty  purpose,  that  we  were  part  of  something 
worthwhile  and  exciting,  and  that  government  could 
be  activist  and  have  a  positive  role  with  health  care 
in  the  country. 

I  have  a  recent  history  in  this  area  as  well, 
although  it  is  informed  by  that  past.  First,  with  the 
American  College  of  Physicians,  I  have  been 
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involved  in  technology  assessment  and  the  produc- 
tion of  clinical  guidelines — standards  for  health 
care — under  our  Clinical  Efficacy  Assessment 
Project.  That  role  has  taught  me  that  professional 
societies  have  a  broader  role  than  merely  looking 
after  the  economic  and  political  power  of  our 
membership;  we  must  work  in  collaboration  with 
the  people  who  are  developing  health  policy. 

Second,  I  have  been  involved  more  recently  in 
the  Robert  Wood  Johnson  Foundation  Hospital 
Quality  Improvement  Program  where  we  are  look- 
ing at  how  hospitals  can  work  cooperatively,  both 
with  each  other  and  with  the  medical  staffs.  One  of 
the  things  I  have  learned  from  reviewing  over  100 
applications  for  the  project  is  that  the  state  of  QA 
in  this  country  is  not  very  good.  There  is  no  cooper- 
ation between  hospital  administration  and  medical 
staff,  or  among  hospitals,  and  there  is  certainly 
no  sharing  of  the  kind  of  data  that  consumers, 
customers,  and  others  need  to  make  appropriate 
choices  in  health  care. 

Third,  I  have  been  a  judge  for  the  Health  Care 
Forum/Witt  Quality  Award,  the  commitment  to 
quality  award  for  hospitals  and  institutions,  and 
I  have  reviewed  a  number  of  applications  from 
people  who  think  they  qualify  for  the  award.  Again, 
the  disappointment  is  that,  even  in  the  best  of 
hospitals,  medical  staffs  are  operating  in  QA 
separately  from  patient  care  QA  and  certainly 
apart  from  the  kind  of  QA  that  the  administrative 
and  management  side  of  the  hospital  wants  to 
implement.  One  of  our  most  important  challengers 
is  to  bring  these  groups  together  for  patient  QA 
rather  than  medical  QA  alone. 

Finally,  I  have  been  on  the  board  of  a  hospital, 
but  like  all  other  hospitals  in  this  country,  QA  was 
looked  at  by  the  medical  staff  as  something  periph- 
eral, not  central,  to  their  activities;  we  need  to 
change  that.  These  experiences  have  led  me  to 
certain  biases  that  will  come  out  in  this  discussion. 

Back  in  the  fall  of  1991,  when  the  Agency  for 
Health  Care  Policy  and  Research  (AHCPR) 


115 


organized  this  conference,  they  asked  me  to  do  a 
wrap-up  on  the  last  day  of  the  meeting.  More 
recently,  they  asked  me  to  present  my  views  on  the 
domains  of  the  public  and  private  sectors  in  this 
area  and  for  my  thoughts  about  how  AHCPR  and 
other  Federal  agencies  can  complement  efforts  by 
the  medical  profession  and  other  private  sector 
groups  to  improve  the  science  and  practice  of  QA. 
They  also  asked  for  my  perspective  on  the  ways  in 
which  medical  effectiveness  and  outcomes  research, 
as  well  as  general  health  services  research,  can  be 
coordinated  with  quality  improvement  (QI)  more 
effectively.  Finally,  they  asked  for  my  views  on 
related  policy  concerns  including  the  potential  role 
of  AHCPR  in  health  care  reform  initiatives.  I  will 
try,  given  my  biases  and  perspectives,  to  summarize 
the  themes  that  emerged  during  this  conference  and 
to  identify  some  of  the  thematic  areas  that  I  would 
like  to  see  the  AHCPR  develop. 

Conference  Summary 

Helen  Smits  (in  this  volume)  set  the  theme: 
"There  is  more  to  health  care  policy  than  cost 
containment"  and  suggested  that  we  all  look  at  the 
larger  policy  context  and  the  real  goals  of  health  care 
policy.  She  listed  them  as  access,  cost,  hassle,  and 
manpower.  Then  she  gave  us  the  litmus  test:  Demo- 
crats mention  access  first,  Republicans  mention  cost 
first.  I  throw  my  vote  in  on  the  access  side.  She  also 
said,  "be  aware  and  beware  of  the  barriers  to  moving 
to  continuous  quality  improvement  and  quality 
assurance,"  two  of  which  I  mention  again:  rigid 
professional  disciplines  and  the  outsider  status  of  the 
medical  staff.  I  think  we  need  to  bring  discussion  of 
these  items  back  into  QA  and  QI  within  the  hospital. 

Just  before  I  began  my  presentation,  someone  in 
the  audience  handed  me  a  note  that  said,  "There 
really  are  only  two  questions:  (1)  Does  it  work? 
and  (2)  If  so,  why?" 

Sheldon  Greenfield  (in  this  volume),  in  talking 
about  the  medical  outcome  studies,  raised  a  point  we 
need  to  listen  to  more  often:  "A  health  service  is  not 
a  health  service,"  it's  part  of  a  package  that  involves 
both  caregivers  and  the  person  receiving  care.  One 
needs  to  take  into  account  not  merely  the  service  but 
also  the  caregiver  and  the  person  receiving  care. 

Heather  Palmer  (in  this  volume)  gave  a  fasci- 
nating taxonomy  of  where  we  have  gone,  from  the 
professional  model  to  the  bureaucratic  model  to 
the  industrial  model.  She  did  not  conclude  like 


many  other  economists  by  saying  that  all  you  need 
to  do  is  change  how  you  pay  and  how  much,  and 
the  health  care  problem  would  be  solved.  Instead, 
she  said,  "What  we  need  to  do  is  to  combine  the 
best  of  care  with  the  best  of  science  and  manage- 
ment." I  think  that  is  our  real  goal  here — a 
transformation  from  the  adversarial  system  we 
now  have  to  one  based  more  on  concepts  of  trust, 
collaboration,  cooperation,  and  the  complementary 
nature  of  the  systems  of  care. 

Finally,  there  were  two  questions  from  the  audi- 
ence that  had  meaning  for  me:  What  are  we  really 
about  here?  What  is  this  alphabet  of  TQM,  CQI, 
QI,  and  QA?  That  should  remind  us  of  what  Paul 
Lembke  and  Beverly  Paine  did  in  the  1940s.  They 
said,  "Before  you  begin  the  medical  audit  for  dis- 
ease make  sure  in  fact  the  patient  had  the  disease 
you  are  auditing  for."  The  second  question  that 
came  from  the  audience  was,  "Does  all  this  outcome 
research,  effectiveness  research,  guidelines,  and  so 
forth  make  a  difference?" 

How  do  I  combine  my  bias  and  perspective  with 
what  I  think  I  heard?  I  believe  it  merges  into  one 
basic  theme:  the  patient-physician  interaction  is 
critical.  The  principal  focus  of  all  that  AHCPR 
does  should  be  to  improve  continuously  the  quality 
of  that  interaction.  I  do  not  mean  to  ignore  other 
perspectives,  but  I  intend  to  use  the  patient- 
physician  interaction  as  the  model  from  which  the 
others  can  be  extrapolated. 

Continuous  quality  improvement  (CQI)  begins 
with  a  customer  and  what  the  customer  wants  and 
needs.  I  believe  that  the  principal  customer  of  CQI 
or  the  health  system  is  not  the  payer,  it  is  the 
patient.  It  is  not  even  the  provider,  who  in  my  view 
is  secondary,  and  it  certainly  is  not  government  and 
other  purchasers.  It  is  first  and  foremost  the  patient. 

CQI  also  says  we  have  to  empower  the  partici- 
pants in  the  system,  and  that  means  we  need  to 
empower  patients  first.  Physicians  and  other  profes- 
sionals also  need  to  be  empowered — not  in  the 
political  and  economic  way,  but  to  improve. 

Finally,  CQI  says  we  have  to  provide  standards 
of  measurement  and  feedback  via  monitoring. 
AHCPR  is  in  a  good  position  to  be  a  catalyst  for 
CQI,  and  it  should  take  the  risk  to  do  so.  The  list 
of  research  questions  presented  at  the  conference 
looks  like  a  list  Congress  might  ask  at  a  hearing.  I 
believe  we  need  to  return  to  very  basic  principles 
and  take  very  basic  kinds  of  risk.  How  do  we  do 
that?  Responding  to  the  charge  of  Helen  Smits, 


116 


I  would  argue  the  goals  of  health  policy  are  very 
simple:  appropriate  health  outcomes  from  appropri- 
ate health  services  for  an  appropriate  price  in  the 
context  of  societal  agreement  on  how  much  we  are 
willing  to  spend. 

AHCPR's  Role 

I  think  the  first  thing  AHCPR  needs  to  do  is 
make  a  clear  commitment  to  its  mission — in  my 
view  to  become  the  primary  quality  information 
resource — by  developing  information  on  health 
outcomes  and  the  effectiveness  of  health  services. 
Second,  to  improve  health  outcomes,  AHCPR 
should  develop,  with  input  from  caregivers  and 
patients,  guidelines  of  clinical  appropriateness  and 
the  least  intrusive  monitoring  systems  and  feedback 
mechanisms  for  continually  improving  health  ser- 
vices. Third,  I  would  argue  that  AHCPR  has  a  basic 
mission  to  catalyze  societal  discussion  about  value 
by  including  all  segments  of  society:  physicians  and 
all  other  health  caregivers,  patients,  payers,  and  all 
others.  I  think  AHCPR  should  focus  on  the  quality 
of  the  process  and  not  worry  so  much  about  QA 
mechanisms.  In  QI,  the  participants,  patients,  and 
physicians  have  the  incentive  to  be  the  primary 
actors,  while  in  QA  the  participants  are  secondary. 
In  QI  they  become  a  collaborative  team.  I  suggest 
focusing  on  the  service  end  of  QI,  not  the  oversight 
end  of  QA.  How  could  that  be  done?  I  think  there 
are  four  basic  activities  AHCPR  might  accomplish: 

1.  Work  on  appropriate  outcomes. 

2.  Work  on  appropriate  services. 

3.  Work  on  catalyzing  discussions  of  societal  value. 

4.  Act  to  make  information  manageable. 

Outcomes.  First,  AHCPR  should  begin  work  on 
appropriate  outcomes.  In  addition  to  continuing  its 
research  on  clinical  effectiveness  and  outcomes, 
AHCPR  should  listen  to  the  customers,  that  is,  the 
patients.  Patients'  values  differ  from  those  of 
researchers,  physicians,  or  hospital  administrators.  If 
you  doubt  that,  consider  what  the  Dartmouth  videos 
are  beginning  to  teach  us.  Remember  what  the 
Yankelovich  survey  from  a  year  ago  said  about  the 
difference  between  the  ways  the  lay  public  and  the 
experts  look  at  health  care  problems.  Physicians  who 
doubt  that  there  are  differences  should  think  about 
what  they  value  in  the  physician-patient  encounter 
compared  with  what  they  value  as  physicians. 


Patients  have  different  values  than  providers, 
and  we  need  to  listen  to  them  as  customers.  AHCPR 
needs  to  work  with  the  public  and  with  professional 
groups  to  do  research  in  four  particular  areas 
related  to  appropriate  outcomes:  first,  research  on 
patient  values  (what  patient  and  physician  charac- 
teristics make  a  difference  in  care);  second,  why 
certain  services  lead  to  certain  outcomes;  third,  the 
linkage  between  outcomes  and  process;  and  fourth, 
ways  in  which  the  appropriate  and  least  intrusive 
systems  possible  can  monitor  what  occurs. 

Appropriate  services.  Let  us  accept  the  fact  that 
physicians  are  good  people  and  intend  to  do  the 
best  for  their  patients.  Rather  than  try  to  identify 
the  "bad  apples"  or  the  "rotten  eggs,"  we  should 
assume  that  most  people  want  to  do  well  and  help 
them  do  so. 

What  must  physicians  and  patients  do  to  provide 
the  best  services  in  the  clinical  situation?  I  would 
argue  they  need  four  things:  valid  information,  a 
credible  source  of  information,  an  incentive  to  use 
the  information,  and  a  safe  environment  in  which 
to  use  the  information. 

Simply  because  we  do  not  have  perfect  informa- 
tion everywhere  does  not  mean  we  should  not  move 
forward.  Valid  information  is  simply  the  best  infor- 
mation developed  using  the  best  existing  data  and 
the  best  expert  opinion.  We  can  begin  developing 
guidelines  for  appropriate  clinical  care  based  on  that 
information  rather  than  waiting  for  perfect  answers. 

What  is  a  credible  source  for  information?  The 
payers  are  not  credible  sources  of  clinical  guidelines, 
nor  are  most  professional  societies,  but  together  we 
ought  to  be  able  to  come  up  with  credible  informa- 
tion. Instead  of  having  self-interest  rule,  we  need 
real  collaboration.  I  see  AHCPR  as  the  perfect 
institution  to  develop  that  collaboration,  and  I 
applaud  what  it  has  already  done  in  this  area. 

In  my  opinion,  AHCPR  does  not  have  a  direct 
role  in  establishing  particular  financial  incentives 
to  use  information.  But  I  do  believe  AHCPR  might 
appropriately  educate.  Research  is  like  the  tree 
falling  in  the  forest;  without  educating  the  people 
who  will  use  it,  the  information  is  useless. 

Physicians  need  a  safe  environment  in  which  to 
use  information.  Physicians  and  others  will  not 
participate  in  technology  assessment,  guidelines 
development,  QA,  or  QI  if  they  perceive  that  the 
information  will  be  used  to  punish  them.  We  need 
to  create  environments  where  the  emphasis  is  on 
improving  quality  of  care  rather  than  punishing  and 
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sanctioning  people.  The  role  of  AHCPR,  in  my 
view,  should  be  to  develop  valid  information,  work 
cooperatively  as  a  credible  source,  and  stimulate  the 
development  of  safe  environments. 

Foster  discussion  of  societal  values.  AHCPR 
should  involve  broad  segments  of  society  in  its 
initiatives.  Outcomes  and  effectiveness  research  and 
evaluation  should  include  not  just  researchers  or 
providers  or  payers,  but  also  the  customers/patients. 
AHCPR  needs  to  involve  patients  in  guidelines 
development  as  well.  For  example,  what  outcomes 
are  valued?  How  can  they  be  taken  into  account? 
Finally,  AHCPR  should  involve  customers  and /or 
patients  in  medically  effective  payment  policy. 

In  reality,  all  existing  and  proposed  benefit 
packages  in  health  care  systems  reform  are  for  the 
benefit  of  the  payer;  they  list  things  that  will  not 
be  reimbursed.  What  if  we  were  to  design  a  benefit 
system  to  benefit  the  patient — for  example,  the 
patient  who  might  need  more  than  100  hospital 
days?  Why  should  we  cut  it  off  at  100  days?  A 
patient  might  need  more  than  10  visits  to  the 
psychologist — why  do  we  cut  it  off  at  that?  How 
could  we  design  a  benefit  package  from  the 
perspective  of  the  patient?  AHCPR  could  be  active 
both  in  the  process  and  the  policy  of  bringing  that 
about,  involving  patients  in  designing  a  medically 
effective  benefit  package. 

In  working  to  catalyze  discussions  of  societal 
values,  AHCPR  needs  to  stimulate  the  educated 
customer  and  the  patient.  In  one  of  the  work  groups 
on  patients,  Lisa  Sullivan  talked  about  the  work 
she  and  Sheldon  Greenfield  did  in  the  medical  out- 
comes study  whereby  patients  were  empowered, 
not  so  much  with  clinical  information — patients 
can  never  know  all  the  clinical  information  that 
physicians  know — but  they  were  empowered  so 
they  could  ask  the  right  questions  and  have  the 
tools  to  negotiate.  AHCPR  could  fill  an  important 
role  in  helping  patients  become  partners  in  care  by 
teaching  them  how  to  talk  to  physicians  and  how  to 
become  partners  in  decisions  about  their  care. 

At  the  American  College  of  Physicians  we  are 
publishing  a  health  letter  that  contains  information 
patients  need,  but  we  do  not  send  it  to  patients;  we 
send  it  to  physicians  to  teach  them  how  to  teach 
their  patients.  AHCPR  could  be  very  effective  in 
enhancing  the  participation  and  interaction  between 
physicians  and  patients  and  by  stimulating  the  edu- 
cated customer /patient. 


Also,  AHCPR  should  consider  education  of  the 
media.  Now,  patients  are  educated  by  the  media  in 
scary  ways,  in  National  Enquirer  "miracle  cure" 
style,  or  through  the  reporting  of  so  much  informa- 
tion that  they  are  overwhelmed.  Clearly,  there  is  a 
role  for  government  here.  Since  I  do  not  see  it 
happening  elsewhere,  perhaps  AHCPR  could  help 
to  educate  the  media  on  how  to  educate  the  public. 

I  would  argue  that  AHCPR  should  work  with 
payers — not  in  the  sense  of  giving  them  information 
about  which  services  they  are  paying  for  or  helping 
them  make  those  decisions — but  by  helping  them  to 
translate  clinical  guidelines  appropriately  into  pay- 
ment guidelines  and  from  there  into  review  criteria. 
If  payers  take  the  clinical  guidelines  without  the 
ability  to  translate  them,  there  could  be  a  disjuncture 
between  clinical  guidelines  and  payment  guidelines. 
Clearly  this  can  be  done  with  the  Health  Care 
Financing  Administration,  and  it  also  might  be 
accomplished  with  other  payers  within  the  system. 

Making  information  more  manageable.  Each 
year  4  million  biomedical  articles  are  published 
in  29,000  journals  worldwide.  That  is  too  much 
information.  AHCPR  could  have  a  role  in  collect- 
ing, analyzing,  distilling,  and  disseminating  usable 
and  relevant  information  to  practitioners.  In  the 
private  sector,  the  American  College  of  Physicians 
publishes  a  periodical  called  ACP  Journal  Club  six 
times  a  year.  This  is  a  survey  of  40  biomedical  jour- 
nals, the  ones  we  think  are  the  best.  Six  times  a  year 
we  pick  the  32  most  important  articles  from  those 
journals  and  publish  a  structured  abstract  of  each 
article  with  a  one-page  critique.  Granted  it  is  the 
USA  Today  of  medical  literature,  but  we  have 
received  kudos  for  it  from  our  members,  who  say 
that  it  has  enabled  them  to  sense  that  they  can  in 
fact  get  their  arms  around  internal  medicine. 
AHCPR  has  a  role  in  collecting,  analyzing,  distill- 
ing, and  disseminating  information.  That  could  be  a 
practical  benefit  in  the  physician/patient  interaction. 

AHCPR  has  a  great  practical  role  in  educating 
management,  particularly  in  quality  management,  at 
least  at  the  medical  school  level  (by  helping  the  edu- 
cators educate  in  the  appropriate  use  of  technology) 
and  certainly  with  continuing  medical  education.  As 
pointed  out  during  this  conference,  physicians  do  not 
know  much  about  quality  management,  and  it  is  the 
rare  physician  who  is  really  trying  to  do  it.  Frankly,  it 
is  the  best  way  for  physicians  to  go. 

It  is  interesting  to  go  to  a  hospital  mortality  or 
morbidity  conference  and  hear  the  discussion.  The 
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purpose  of  the  discussion  is  to  find  out  whether  God 
was  at  fault  ("God's  will"),  or  the  physician  was  the 
culprit.  That  does  not  teach  anything  because  the 
great  incentive  when  those  are  the  only  two  options 
is  to  blame  God.  What  we  should  be  doing  is  look- 
ing at  events  and  trying  to  find  out  how  can  we  learn 
from  them  and  make  improvements.  No  one 
teaches  that  well.  AHCPR  has  an  important  role  in 
making  the  information  manageable,  particularly 
information  about  quality  management  and  CQI. 

Role  of  Professional  Societies 

What  is  the  professional  society's  role  in  inter- 
acting with  AHCPR?  It  comes  down  to  one  word: 
leadership.  Professional  societies  finally  need  to 
take  the  risk  for  societal  gain  in  two  areas:  one,  to 
set  standards  cooperatively  with  AHCPR;  and  two, 
to  cooperate  not  only  with  AHCPR  but  also  with 
patients,  payers,  labor,  consumers,  and  industry  to 
try  to  reform  the  health  care  system.  This  will  not  be 
accomplished  by  all  of  us  taking  our  separate  posi- 
tions and  fighting  about  them.  We  have  to  come 


together  collaboratively,  and  unless  professional 
societies  take  the  lead,  I  do  not  believe  we  are  going 
to  get  anywhere.  Unless  we  do  this,  we  will  not  be 
sending  the  right  message  to  practicing  physicians. 

Conclusion 

In  summary,  AHCPR  will  have  a  central  role  in 
any  reformed  health  care  system  of  the  future 
because  there  will  always  be  greater  demands  for 
health  services  than  resources  available  to  meet 
them.  Societal  choices  must  therefore  be  based  on 
attempting  to  improve  health  care  outcomes  by 
continually  improving  health  services.  By  focusing 
on  the  patient-physician  interface  and  viewing  the 
informed  and  empowered  patient  as  the  customer, 
we  have  the  best  chance  of  reaching  the  goal  of 
improved  health  care. 

In  my  view,  QA  needs  to  be  transformed  into 
QI.  If  it  is,  then  a  partnership  can  and  should 
develop — not  an  assessment  for  punitive  sanction, 
but  a  cooperative  evaluation  to  improve  care.  This 
is  different  in  concept,  difficult  to  achieve,  but 
critical  to  begin. 
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Summary  Presentation 


Closing  Quality  Improvement/Quality  Assurance  Information  Gaps: 
Public  and  Private  Sector  Roles 


David  N.  Sundwall,  M.D. 


Introduction 

I  compliment  J.  Jarrett  Clinton  and  the  Agency 
for  Health  Care  Policy  and  Research  for  sponsoring 
this  conference.  I  see  AHCPR  as  a  complementary 
force  to  the  Health  Care  Financing  Administration 
(HCFA)  in  assisting  those  of  us  working  in  the  pri- 
vate sector.  I  welcomed  this  assignment  because  it 
gave  me  a  chance  to  focus  on  the  impressions  I  have 
had  for  some  time  about  what  is  right  and  wrong  in 
hospital  quality  improvement  (QI)  efforts,  and  like 
John  Ball,  I  am  going  to  talk  from  the  point  of  view 
of  a  physician,  but  one  who  for  3-1/2  years  has  been 
involved  with  the  not-for-profit  hospital  community. 
I  apologize  to  others  who  have  a  different  perspec- 
tive, for  example,  home  care,  or  nursing  care,  or 
other  areas  that  are  equally  important. 

American  Health  Care  Systems  (AmHS) 

I  will  take  a  moment  to  describe  my  organiza- 
tion and  why  we  care  about  quality  in  health  care. 
American  Health  Care  Systems  (AmHS)  was 
established  about  a  decade  ago  as  an  alliance  of 
multiple  hospital  systems.  We  have  40  systems, 
representing  1,100  facilities  in  47  States.  Their 
annual  revenues  total  $22.5  billion,  and  they  consti- 
tute about  one-fifth  of  the  Nation's  not-for-profit 
hospital  beds.  These  40  hospital  systems  all  have  a 
not-for-profit  mission,  and  most  of  them  have  a 
religious  heritage.  Their  mission  is  to  serve  their 
communities,  and  the  alliance  exists  to  make  them 
more  efficient  and  economical. 

The  credo  of  AmHS  is  to  assist  our  member 
hospitals  in  providing  the  highest  quality  of  health 
care  at  the  lowest  possible  cost.  We  do  this  through 
a  broad  range  of  services,  and  this  includes  the 
promotion  of  quality.  I  say  promotion  of  quality, 
because  as  an  alliance,  we  can  cajole,  persuade,  and 
encourage  our  member  hospitals  that  the  best  care 
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possible  is  good  for  their  bottom  line  and  competi- 
tiveness, as  well  as  important  to  their  missions,  but 
we  do  not  have  any  leverage  over  them.  The  follow- 
ing are  some  of  the  things  AmHS  does  to  promote 
high  quality  care. 

Education.  In  June  1989  we  began  publishing 
The  Quality  Agenda,  an  annually  updated  mono- 
graph that  provides  an  overview  of  who  is  doing 
what  with  regard  to  health  care  quality  issues.  It  is 
hardly  a  comprehensive  list  or  directory,  but  it 
provides  a  broad  overview  of  quality-related 
activities  being  undertaken  by  the  Federal  and  State 
governments,  business  coalitions,  the  Institute  of 
Medicine,  academic  groups,  and  other  interested 
parties.  We  did  this  because  it  helps  sort  out  multi- 
ple activities  and  helps  our  hospitals  anticipate  new 
developments.  We  also  make  clear  that  quality 
should  be  understood  as  documentation  of  value, 
accountability,  and  public  disclosure. 

Another  annual  activity  is  our  sponsorship  of 
a  QI  education  conference.  We  focus  on  quality- 
related  national  health  policy  and  trends,  and  we 
provide  an  opportunity  to  highlight  AmHS  quality 
activities  that  have  been  successful.  We  schedule 
national  leaders  to  tell  us  of  their  particular  activi- 
ties— for  example,  Dr.  Clinton  spoke  a  few  years 
ago,  and  last  year  we  heard  from  Nancy  Gary,  the 
physician  advisor  to  HCFA.  At  the  request  of  the 
Board  of  AmHS  we  also  make  our  conferences 
clinically  oriented.  They  have  been  fairly  successful 
in  having  management  embrace  the  concepts  of 
continuous  quality  improvement,  but  they  have 
been  less  successful  with  engaging  medical  and 
nursing  staffs  in  «uch  efforts.  Periodically  we  also 
publish  other  educational  materials  that  are  devoted 
to  quality  issues. 

Research.  We  also  conduct  research  at  the 
AmHS  Institute.  For  example,  we  did  a  recent  study 
in  collaboration  with  the  Johns  Hopkins  Center  for 
Hospital  Finance  and  Management  on  excess  hospi- 
tal capacity,  in  which  we  concluded  that  there  are 
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approximately  200,000  excess  acute  care  beds  in  the 
United  States.  We  did  another  analysis  comparing 
not-for-profit  hospital  tax  benefits  with  their  contri- 
butions to  their  respective  communities.  This  raises 
interesting  policy  issues,  because  while  the  majority 
of  our  systems  collectively  provided  more  commu- 
nity service  than  they  gained  from  the  government 
tax  exemption,  some  systems  do  not.  Consequently, 
we  will  be  repeating  the  survey,  with  a  goal  of 
establishing  a  standard  of  community  service  that 
all  of  our  members  should  strive  to  achieve. 

As  part  of  our  research  effort,  we  have  partici- 
pated as  a  charter  member  of  the  Quality  Measure- 
ment and  Management  Project  (QMMP),  run  by  the 
Hospital  Research  and  Education  Trust  (HRET),  a 
4-year  project  scheduled  for  completion  in  the  sum- 
mer of  1992.  Its  intent  is  to  demonstrate  leadership 
among  the  hospital  community  by  developing  scien- 
tifically sound  comparative  hospital  outcome  studies 
based  on  available  hospital  data  and  promoting 
nationwide  use  of  a  simplified  uniform  method  of 
hospital  quality  management.  We  published  a  series 
of  monographs  that  are  helpful  and  established  a 
dialogue  with  several  nonhealth  entitles  interested 
in  hospital  quality  data,  such  as  businesses,  the 
academic  community,  employers,  and  regulators. 
Some  attending  this  conference  have  served  on  our 
external  advisory  group  or  as  technical  advisors. 

Discounted  contracts.  Because  we  are  in  the 
business  of  providing  savings  for  our  members,  we 
seek  favorable  contracts  for  a  variety  of  goods  and 
services  that  are  hospital  quality  related.  In  return 
for  a  significant  discount,  generally  based  on  the 
number  of  our  hospitals  participating,  we  provide 
introduction  and  opportunities  for  vendors  to  edu- 
cate about  the  potential  merits  of  their  products 
which  are  designed  to  improve  the  quality  of 
hospital  care. 

Hospital  Quality:  Where  Are  We  in  the 
Revolution? 

Now  I  will  share  with  you  some  personal 
impressions,  as  John  Ball  did  from  his  perspective. 
I  have  been  working  for  over  3  years  in  the  not-for- 
profit  hospital  community,  and  it  has  been  a  fasci- 
nating education  for  me.  I  have  learned  to  walk  in 
another  person's  moccasins  because  I  came  to  my 
current  position  with  some  of  the  same  prejudices 
that  John  described  regarding  the  conflicts  between 
the  hospital  and  the  doctor.  The  interface  between 


hospitals  and  doctors  often  is  tense  and  trouble- 
some, reflecting  what  are  considered  competing 
and  conflicting  interests. 

Positive  developments.  The  following  are  some 
of  my  positive  impressions.  One  is  that  multiple 
efforts  from  a  variety  of  sources  (many  of  them  very 
creative  and  innovative),  both  internal  and  external 
to  hospitals,  are  directly  affecting  the  quality  of  care 
provided  in  hospitals  and  collectively  are  making 
a  difference.  Please  do  not  ask  me  for  my  measure- 
ment instrument;  I  come  to  this  conclusion  from  a 
growing  list  of  anecdotal  reports  from  a  number  of 
our  member  hospitals  that  cite  improving  efficiency, 
decreasing  cost,  better  morale  for  their  providers, 
and  more  patient  satisfaction.  However,  valid 
measurement  tools  are  wanting,  and  many  of  these 
results  are  self-reported. 

Second,  through  the  wonders  of  computers, 
large  data  bases  have  in  recent  years  given  us  the 
wherewithal  to  make  clinical  decisions  based  not 
just  on  our  colleagues  or  the  CME  course  we  went 
to  last  year  but  on  thousands  of  patient  encounters. 
This  is  a  real  plus,  and  I  think  there  is  a  lot  of 
promise  that  such  data  systems  can  provide  valuable 
tools  for  much  improved  quality  of  patient  care. 

My  third  impression  is  that  there  has  been  more 
attention  on  health  service  research,  which  is  long 
overdue.  I  now  have  a  decade  of  experience  at  the 
Federal  level  in  health  policy,  and  while  I  enjoyed 
the  oversight  hearings  on  the  National  Institutes  of 
Health,  which  focused  on  biomedical  research,  I  was 
increasingly  pained  by  the  relative  lack  of  support 
for  health  services  research.  Fortunately,  more 
attention  is  being  paid  to  such  research  because  of 
the  recent  focus  on  outcomes  and  practice  variation. 
It  is  appropriate  and  most  welcome. 

A  fourth  impression  is  what  I  call  the  medical- 
ization  of  QI.  I  do  not  want  to  limit  this  to  physi- 
cians, because  increasing  numbers  of  physicians  and 
nurses  are  becoming  involved  in  QI  at  the  hospital 
level,  as  well  as  at  the  corporate  level  of  hospital 
systems.  This  is  an  encouraging  development.  It 
holds  the  promise  of  identifying,  in  a  way  we  have 
not  previously  done,  genuine  common  interests 
among  the  hospital  managers  and  doctors,  enabling 
the  bonding  that  is  frequently  talked  about  and 
diminishing  the  differences  and  barriers  that  get  in 
the  way.  When  clinicians  are  given  quality-related 
information,  they  generally  welcome  a  chance  to 
wrestle  with  data  and  figure  out  how  to  make  better 
choices  and  clinical  improvements. 
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The  fifth  observation  is  the  constructive  new 
agenda  of  the  Joint  Commission  on  Accreditation  of 
Healthcare  Organizations.  I  think  it  is  an  important 
step  forward.  They  have  fostered  a  lot  of  participa- 
tion among  physicians  in  the  development  of  their 
new  quality  indicators.  I  realize  we  can  quibble 
about  the  pace  at  which  this  is  progressing,  but  at 
the  same  time  it  is  movement  in  the  right  direction. 

Possible  problems.  The  following  are  some  neg- 
ative impressions  of  the  quality  movement,  concerns 
that  were  reinforced  at  this  conference.  I  will  start 
with  what  I  call  the  commercialization  of  QI.  As  we 
have  an  opportunity  to  listen  to  the  vendors  of 
goods  and  services  and  consultants,  I  am  amazed  at 
how  many  firms  are  getting  into  the  business  of 
assisting  our  hospitals  with  QI  and  have  developed 
products  or  services  costing  hundreds  of  thousands 
of  dollars.  I  am  concerned  because  there  has  been 
little  validation  and  testing  of  many  of  these  prod- 
ucts and  services.  We  advise  our  members  to  make 
sure  that  the  quality-related  consultation,  goods,  or 
services  are  appropriate  for  their  institution;  they 
practice  due  diligence  before  making  a  decision  to 
purchase;  and  they  have  the  necessary  trained  per- 
sonnel to  ensure  the  desired  outcome. 

Second,  I  am  convinced  we  are  at  the  beginning 
of  a  "quality  race,"  akin  to  the  technology  race, 
where  hospitals,  fearing  they  are  losing  their  market 
or  not  being  as  aggressive  as  the  hospital  down  the 
street,  are  embracing  untried  and  unproven  prod- 
ucts in  an  attempt  to  keep  pace.  This  quality  race 
can  impede  success  because,  in  their  impatience  to 
get  on  with  it,  they  have  neither  laid  the  necessary 
groundwork  nor  invested  in  sufficient  training  or 
personnel.  For  example,  although  I  love  the  Nike 
approach  of  "just  do  it,"  when  it  comes  to  QI,  impa- 
tience and  inadequate  emphasis  on  infrastructure, 
resources,  and  personnel  can  compound  problems. 

My  third  negative  impression  is  the  problem 
caused  by  the  demands  for  data  placed  on  hospitals. 
Later,  I  will  discuss  what  we  are  trying  to  do  to  cor- 
rect that,  but  I  can  assure  you  it  is  a  significant  and 
costly  burden.  A  variety  of  organizations  want  to 
assess  how  well  our  hospitals  are  doing,  and  in  the 
process  they  are  surveying  us  to  death.  A  member 
of  the  audience  at  our  annual  conference  in  1991 
vented  her  anger  at  the  HCFA  representative  after 
her  presentation.  She  gave  a  litany  of  seven  surveys 
she  had  been  asked  to  complete  in  the  previous  6 
weeks.  Such  quality-related  surveys  are  often 


cumbersome,  time  consuming,  and  collectively  a 
tremendous  burden  on  our  hospitals. 

The  fourth  problem  I  have  observed  is  the 
prevalence  of  data  illiteracy.  Many  hospital  person- 
nel, expected  to  "get  with  it"  on  quality  initiatives, 
are  simply  not  prepared  to  understand  or  use  many 
of  the  tools  they  are  expected  to  use.  Many  QI  hos- 
pital folks  are  intimidated  and  turned  off  by  compli- 
cated data  management  systems  and  hospital  com- 
parative reports,  and  I  think  more  "user-friendly" 
systems  and  technical  assistance  are  needed. 

Fifth,  while  data  are  not  the  same  as  informa- 
tion, some  purchasers  of  health  care  unfortunately 
do  not  understand  the  difference.  Major  decisions 
about  purchasing  and  contracting  are  being  made 
based  simply  on  data,  not  meaningful  information. 

The  last  issue  I  will  cite  is  one  others  have 
mentioned,  and  that  is  extraordinarily  poor  coordi- 
nation. As  we  undertake  multiple  QI  initiatives,  we 
are  in  danger  of  contributing  to  the  cost  and  ineffi- 
ciency of  hospital  management,  if  all  interested  par- 
ties can't  come  together  and  agree  on  an  appropriate 
data  management  and  measurement  system.  We 
very  much  need  partnerships  of  the  public  and 
private  sector  if  we  are  to  become  more  effective 
and  not  contribute  to  the  problems. 

External  Data  Demands 

For  the  QMMP,  I  chaired  a  subcommittee 
on  external  data  demands.  We  have  produced  a 
monograph  that  captures  some  of  our  thinking 
about  the  multiple  requests  for  hospital  quality  data 
from  many  diverse  organizations.  It  describes  our 
concern  about  the  expense,  confusion,  and  misun- 
derstanding and  the  invalid  assumption  that  might 
be  made  from  the  data  we  provide.  In  short,  we 
desire  to  bring  order  out  of  chaos  and  to  promote 
consensus  among  all  parties:  governments,  purchas- 
ers, payers,  researchers,  and  providers.  What  we  do 
not  want  to  do,  by  any  means,  is  stifle  or  discourage 
requests  for  quality-related  hospital  data,  and  we  do 
not  want  to  give  external  organizations  just  the 
hospital  data  they  need  or  limit  what  we  are  going 
to  give.  We  do  not  think  it  feasible  to  develop  a 
definitive  data  system  that  could  be  applied  to  all 
settings,  but  we  do  want  to  do  the  following  things: 

•  Reach  out  to  those  who  are  interested  in  estab- 
lishing ongoing  collaboration  between  all  inter- 
ested parties  on  hospital  quality  information. 
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•  Develop  consensus  on  uniform  data  manage- 
ment system  (not  the  specific  data  elements). 

•  Establish  a  dialog  between  all  interested  parties 
to  enable  the  revising  and  updating  of  this 
process  over  time.  The  process  should  be 
dynamic. 

•  Apply  the  system  to  hospital  quality  manage- 
ment as  well  as  external  reviewers  to  the  extent 
that  it  is  the  same,  and  saves  redundancy,  and 
would  be  complementary. 

To  do  these  things,  we  documented  the  scope  of 
external  demands  in  an  HRET  publication  titled 
"An  Inventory  of  External  Hospital  Data 
Demands."  This  report  demonstrates  our  burden  in 
trying  to  accommodate  the  external  demands  from 
many  people  and  entities  interested  in  how  well  we 
are  doing.  Second,  we  invited  a  broad  group  of  tech- 
nical advisors  to  meet  with  the  QMMP  task  force  in 
Chicago  to  advise  us  on  what  information  they  felt 
was  essential  to  make  meaningful  judgments.  We 
then  developed  some  guiding  principles  and  a 
conceptual  framework. 

I  have  heard  others  refer  to  the  enormous 
amount  of  available  data  of  questionable  usefulness. 
Dr.  John  Williamson  of  the  Veterans  Administra- 
tion Hospital  in  Salt  Lake  City  has  done  some  inter- 
esting research  on  this  topic.  He  believes  that  from 
the  universe  of  data  which  are  theoretically  useful  in 
measuring  whether  something  is  good  or  bad,  only 
certain  elements  will  prove  really  useful  as  they  are 
subjected  to  a  series  of  relevance  filters  and  applied 
to  testing.  Such  data  elements  must  be  used — that 
is,  they  must  go  through  a  maturity  filter  over 
time — and  they  must  be  subjected  to  scientific  rigor. 
A  limited  amount  of  data  is  left  that  is  replicable, 
valid,  and  useful  across  the  board. 

We  are  in  a  primitive  state  with  data  manage- 
ment; I  am  afraid  that  many  of  us  are  still  floating  in 
a  sea  of  data,  much  of  which  is  not  likely  to  be  very 
helpful.  An  important  part  of  the  quality  research 
agenda  will  be  to  help  validate  some  of  these  data 
elements.  Our  QMMP  committee  concluded  that 
there  are  four  categories  of  hospital  quality  data 
that  are  worth  collecting,  and  if  we  can  get  others 
to  think  according  to  these  categories,  we  can 
develop  a  simplified  and  more  useful  system  of 
quality  data  management  and  reporting.  The 
categories  are:  appropriate  treatment  indications 
for  major  process  factors;  medical  outcomes,  which 
include  functional  and  patient  satisfaction;  price, 
charges  and  costs;  and  practitioners'  satisfaction 


(we  believe  the  morale  of  providers  is  an  important 
indicator  of  the  quality  of  care). 

AmHS  Quality  Initiatives 

AmHS  will  proceed  with  two  quality-related 
research  projects  in  the  near  future: 

1.  A  feasibility  study  of  AmHS  maintaining  a 
centralized  QI  data  monitoring  system  for  our 
member  hospitals.  We  have  selected  the 
PRAGmatic  System  (developed  by  Corporate 
Cost  Management  in  Rockville,  Maryland)  for 
testing,  using  a  sophisticated  analysis  of  data 
elements  obtained  from  the  Uniform  Discharge 
Data  Set,  to  give  us  four  indicators  (mortality, 
long  length  of  stay,  short  length  of  stay,  and 
complications)  for  882  risk  categories. 

2.  Clinical  benchmarking  for  MedisGroup  hospi- 
tals. We  have  several  hospitals  that  are  using 
MedisGroup  software  systems,  some  because 
they  are  required  to  (in  the  States  of  Iowa  or 
Pennsylvania)  and  some  because  they  have  cho- 
sen on  their  own  to  purchase  the  MEDIQUAL 
products.  We  will  assist  our  hospitals  using 
MedisGroup  data  to  define  what  we  call 
clinical  benchmarks  (that  is,  best  practices) 
that  complement  clinical  practice  guideline 
development  but  are  hospital-specific. 

Conclusion 

This  conference  provided  a  welcome  focus  on 
health  care  quality-related  research,  and  it  was  a  real 
service  for  the  hospital  community.  We  all  have  a  lot 
to  do  to  improve  our  country's  health  care  system. 

I  will  close  with  a  plea  for  balance  in  research 
investment  in  basic  science,  such  as  improving  the 
science  of  data  management  and  developing  indica- 
tors, and  investing  in  applied  research.  Please  help 
us  develop  some  user-friendly  tools  as  soon  as  possi- 
ble. From  my  perspective,  we  are  in  the  midst  of  a 
revolution  in  health  care  quality — its  measurement, 
monitoring,  and  management — but  we  are  very 
early  in  this  process  that  was  catalyzed  by  the  dawn 
of  the  computer  age  and  have  a  long  and  arduous 
struggle  ahead  of  us. 

Reference 

Hospital  Quality  Related  Data:  Recommendations  for 
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Workshop  Recommendations 


Workgroup  I. 

Information  Technologies  and  Quality 
Improvement/Quality  Assurance 

Norman  W.  Weissman,  Ph.D.,  Topic  Reporter 
Workshop:  Automated  Data  Resources 

Facilitator:  John  Kelly,  M.D.,  Ph.D. 
Research  Discussant:  William  Tierney,  M.D. 

Workshop:  Medical  Informatics  and 
Quality  Improvement/Quality  Assurance 

Facilitator:  Norman  W.  Weissman,  Ph.D. 
Research  Discussant:  Mark  Musen,  Ph.D. 


The  work  groups  on  information  technologies 
saw  the  future  of  quality  improvement  (QI)  and 
quality  assurance  (QA)  as  very  closely  tied  to  the 
future  of  information  systems.  They  therefore 
placed  a  high  priority  on  research,  development, 
demonstrations,  and  evaluations  of  the  impact  of 
information  systems  on  quality.  Members  felt  that 
information  systems  will  support  decisionmaking  by 
patients  and  allow  incorporation  of  their  values  into 
the  decisionmaking  process.  It  is  therefore  impor- 
tant to  learn  what  existing  systems  can  do — whether 
at  the  review  level  or  at  the  level  of  patient  system 
supports — and  what  improvements  in  data  analysis 
and  aggregation  can  be  made. 

Finally,  the  group  felt  that  diffusion  was  greatly 
affected  by  factors  other  than  health  care  and 
encouraged  social  science  research  on  factors  lead- 
ing to  the  adoption  and  use  of  information  systems. 
Special  attention  was  directed  to  issues  of  confiden- 
tiality and  privacy. 

Research  needs  were  identified  in  seven  areas 
involving  use  of  automated  data  resources: 

1.  What  are  the  effects  of  automated  information 
systems  on  quality?  Assertions  are  often  made 
about  the  quality-enhancing  abilities  of  elec- 
tronic information  handling  systems,  but  such 
claims  need  evaluation  and  assessment  by 
parties  independent  of  the  developers  and 
marketers  of  such  systems.  This  includes  not 
only  hospital  information  systems,  but  also 
smaller  systems  used  by  individuals. 


2.  How  can  patient-specific  probabilities  and 
utilities  be  incorporated  into  decision  support 
systems?  The  patient  cannot  be  left  out  of 
considerations  of  quality.  Information  systems  in 
general,  as  well  as  high-technology  decision 
support  systems  of  the  future,  can  provide  a  way 
to  include  patients'  concerns  in  decisionmaking, 
but  they  must  permit  improvements  in  quality 
of  care  for  specific  patients. 

3.  How  can  data  in  existing  information  systems 
be  used,  and  what  elements  should  be  added  to 
improve  quality?  Discussion  of  research  needs 
ranged  from  the  reliability,  transportability,  and 
validity  of  currently  collected  information  to  the 
specification  of  additional  elements  for  multiple 
applications. 

4.  What  are  the  sociologic  factors  that  influence 
the  adoption  of  information  systems  technology? 
The  ability  to  use  information  resources  is  very 
dependent  on  organizational  and  environmental 
variables.  Sociologic  research  is  needed  on 
implementation  processes  in  different  settings 
with  different  applications. 

5.  What  are  the  roles  of  technology,  policy,  legisla- 
tion, and  public  opinion  in  solving  the  problems 
of  privacy  and  confidentiality  of  electronic 
records?  Privacy  and  confidentiality  are  major 
concerns  with  electronic  data  systems,  especially 
with  data  collected  longitudinally  which  require 
use  of  identifiers  to  link  successive  records. 

It  was  agreed  that  most  data  suitable  for  use  for 
QI  would  need  to  be  linked.  To  avoid  inappro- 
priate use  of  the  data,  it  was  recommended  that 
technologic  and  legislative  solutions  and  the  util- 
ity of  policies  and  public  opinion  in  influencing 
the  use  of  such  data  be  investigated. 

6.  What  methods  are  appropriate  for  analyzing 
longitudinal  data  sets?  Most  of  the  analytical 
tools  that  have  been  developed  for  health  care 
research  have  been  for  experimental  studies.  It 
was  recommended  that  methodological  research 
be  given  a  high  priority,  with  attention  to  issues 
of  sample  selection,  choice  of  variables,  and 
modeling. 
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7.  What  are  the  most  effective  ways  of  disseminat- 
ing the  systems  that  improve  quality  and  the 
information  derived  from  these  systems?  Many 
discussants  pointed  to  the  lack  of  diffusion  of 
information  about  existing  successes,  as  well 
as  failures,  and  urged  research  to  improve 
dissemination  to  avoid  unnecessary  replication 
of  mistakes. 


Workgroup  II: 

Methods  and  Measures  for  Quality 
Improvement/Quality  Assurance 


Marcia  Orsolitis  Stevic,  R.N.,  Ph.D.,  Topic  Reporter 

Workshop:  Quality  Review  Criteria 

Facilitator:  Carole  Magoffin 
Research  Discussant: 

Marcia  Orsolits  Stevic,  R.N.,  Ph.D. 

Workshop:  Performance  Standards 

Facilitator:  Margaret  O'Kane 

Research  Discussant:  Vahe  Kazandjian,  Ph.D. 

Workshop:  Improving  the  Science  of  Quality 
Improvement/Quality  Assurance 

Facilitator:  Leon  Wyszewianski,  Ph.D. 
Research  Discussant:  Haya  R.  Rubin,  M.D.,  Ph.D. 


The  work  group  on  quality  review  criteria 
demonstrated  an  appreciation  of  the  role  of  scien- 
tific studies  in  the  diverse  arena  of  QI/QA.  A  num- 
ber of  themes  became  apparent  in  the  discussions. 
There  was  concern  about  the  variation  in  needs 
among  health  care  sites  that  may  be  in  different 
stages  of  development  in  their  ability  to  measure 
quality  or  implement  more  sophisticated  methods. 

Issues  involved  in  dealing  with  different  popula- 
tions were  another  concern,  including  the  needs  of 
diverse  ethnic  and  cultural  groups  and  different  age 
groups.  Both  short-  and  long-term  needs  for  QI 
review  criteria  were  examined.  Concern  was 
expressed  about  some  tools  that  may  have  been  too 
quickly  marketed  and  adopted  and  may  now  need 
to  be  re-examined  for  reliability,  validity,  and  ability 
to  give  information  across  sites.  The  relationship 
between  guidelines  and  criteria  was  also  discussed. 

Research  needs  were  identified  in  three  areas  of 
quality  review  criteria: 


1.  What  are  the  best  methods  for  developing, 
implementing,  and  evaluating  review  criteria? 
What  are  the  links  between  process  and  out- 
come? What  are  the  best  methods  of  feedback  to 
change  practice,  and  what  elements  of  practice 
are  most  amenable  to  change?  Discussants  con- 
sidered the  link  between  process  and  outcome 
to  be  a  key  issue. 

2.  What  are  the  best  ways  to  design  and  measure 
guidelines  and  criteria  across  sites  of  care,  taking 
into  account  identification  of  the  locus  of  respon- 
sibility for  decisions?  It  is  sometimes  difficult  to 
identify  who  is  responsible  for  what  portion  of 
care.  As  emphasis  on  a  continuum  of  care 
increases,  this  issue  will  become  more  complex 
and  challenging.  The  issue  is  not  only  who  has 
responsibility  for  what  portion  of  care,  but  also 
who  has  input  to  shape  the  evaluation  criteria 
and  who  receives  information  about  it. 

3.  How  do  national  and  local  criteria  compare? 
How  do  criteria  developed  from  the  generalist 
perspective  compare  with  those  developed  from 
the  specialist  perspective  in  terms  of  acceptabil- 
ity and  usefulness?  It  was  appreciated  that 
acceptability  and  usefulness  are  intertwined; 
people  who  decide  that  some  things  are  unac- 
ceptable are  not  likely  to  regard  them  as  useful, 
although  in  other  arenas  they  might  be. 

Discussants  in  the  workshop  on  quality  stand- 
ards focused  on  such  issues  as:  What  are  the 
methodologies  for  developing  standards?  Who  is 
involved  in  developing  and  implementing  them? 
How  are  standards  updated?  How  long  do  they 
last?  What  are  the  costs  involved?  The  following 
research  questions  were  identified: 

1.  How  can  continuous  quality  improvement  (CQI) 
methods  be  used  to  identify  and  translate  into 
practice  the  key  processes  that  affect  outcomes? 
Discussants  agreed  that  standards  should  be 
based  on  these  key  processes. 

2.  Are  current  protocols  and  guidelines  having  an 
impact  on  quality,  and  what  difference  do  they 
make  in  terms  of  cost  and  patient  outcomes? 
After  a  period  in  which  measurement  of  cost 
was  the  major  focus  there  is  now  interest  in 
measuring  quality,  and  there  is  a  need  to  put 
them  together. 
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3.  In  addition  to  the  medical  record,  what  other 
aspects  of  the  care  process  should  be  captured, 
compared,  and  contrasted  to  improve  the  quality 
of  care?  We  can  no  longer  focus  only  on  the 
medical  record.  Multiple  sources  of  information 
need  to  be  tapped. 

4.  How  does  the  development  and  application  of 
standards  contribute  to  the  adoption  of  stand- 
ards by  organizations,  and  how  do  they  influence 
practice  and  outcomes?  For  example,  does  the 
credibility  of  the  review  by  which  standards  are 
developed  affect  their  success? 

In  the  workshop  on  improving  the  science  of 
QI/QA,  discussants  were  principally  interested  in 
increased  scientific  rigor  in  QI/QA  studies.  The 
following  research  questions  were  identified: 


1.  What  methods  can  be  used  to  improve  the  relia- 
bility and  validity  of  measures  of  quality  in  well- 
controlled  investigations?  Many  measures 
should  be  examined  in  controlled  experiments 
to  determine  whether  they  make  a  difference. 

2.  What  longitudinal  measures  of  quality  of  care — 
including  the  process  of  care — best  represent  the 
patient  episode  regardless  of  setting?  Patients 
can  move  between  care  settings  over  time.  Are 
there  other  sources  besides  the  patients  them- 
selves that  can  provide  usable  information? 

3.  In  the  short  term,  evaluate  and  validate  com- 
monly used  measures,  for  example,  readmissions 
and  outcome  tools. 


Workgroup  III: 

Organization  of  Quality  Improvement/ 
Quality  Assurance  Systems 

David  B.  Nash,  M.D.,  M.B.A.,  Topic  Reporter 

Workshop:  Delivery  Systems 

Facilitator:  Robert  A.  Fried,  M.D. 

Research  Discussant:  Shoshanna  Sofaer,  Dr.P.H. 

Workshop:  Control  and  Responsibility 
for  Quality  Review 

Facilitator:  James  Roberts,  M.D. 
Research  Discussant: 

David  Hsia,  J.D.,  M.D.,  M.P.H. 

Workshop:  Resources  for  Effective  QI/QA 

Facilitator:  David  B.  Nash,  M.D.,  M.B.A. 
Research  Discussant:  Gary  Filerman,  Ph.D. 

The  workgroup  on  delivery  systems  decided  that 
QI,  for  purposes  of  its  discussion,  would  not  neces- 
sarily mean  total  quality  management  (TQM),  CQI, 
or  QA.  Instead,  the  focus  was  on  quality  of  care  in 
general,  by  whatever  mechanism.  There  was  consid- 
erable initial  discussion  about  which  variables  were 
dependent  on  QI  and  which  were  not.  Three  main 
questions  guided  the  discussions:  (1)  Are  we  going 
to  give  QI  a  chance?  (2)  If  so,  is  it  worth  doing? 
(3)  What  are  the  key  elements  to  deal  with  if  we 


choose  to  implement  QI?  Research  questions  were 
identified  in  three  main  areas: 

1.  Organizational  readiness.  This  includes  individ- 
ual variables  related  to  institutional  and  profes- 
sional culture,  which  are  frequently  overlooked; 
organization  and  structure  of  the  medical  staff 
— for  example,  the  presence  or  absence  of  full- 
time  physicians  involved  in  these  activities; 
whether  the  organization  views  QI  as  a  "bottom- 
line"  issue;  and  whether  QI  is  a  reaction  to  a 
crisis  or  the  idea  of  a  chief  executive  officer. 

2.  Implementation,  including:  is  a  certain  kind  of 
organizational  structure  more  readily  able  to 
adopt  QI?  If  QI  is  accepted,  where  is  the  best 
place  to  put  it  in  the  organization  and  with  what 
level  of  support? 

3.  Measuring  the  effects.  Some  organizations  seem 
to  have  incorporated  QI  successfully.  What  are 
the  characteristics  of  those  organizations,  and 
what  lessons  can  be  learned  from  them? 

Discussions  in  the  workshop  on  control  and 
responsibility  for  quality  review  led  to  the  identifi- 
cation of  three  principal  areas  for  research: 

1.  Examination  of  descriptive  and  comparative 
studies  on  Federal  and  State  regulations,  accred- 
itation, and  community-based  efforts  involving 
multiple  payers.  The  goals  should  be  to  deter- 
mine what  is  being  done,  what  accounts  for 
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variations,  what  the  variations  mean,  and  how 
processes  are  changing  over  time.  Discussants 
also  thought  it  would  be  useful  to  examine  the 
regulatory  environment  in  other  fields  and 
identify  lessons  for  health  care. 

2.  What  effects  do  variations  in  processes  have? 
How  do  they  relate  to  population-based  patient 
outcomes,  organizational  efficiency,  and  admin- 
istrative costs? 

3.  What  are  the  effects  of  oversight?  Can  we  iden- 
tify an  optimal  relationship  between  an  external 
review  of  quality  of  care  in  an  institution?  How 
does  external  review  stimulate,  support,  or 
subvert  the  QI  process?  What  is  the  interface 
between  external  review  and  internal  processes? 

In  the  workshop  on  resources  for  effective 
QI/QA,  participants  considered  the  state  of 
knowledge  about  the  human  and  fiscal  resources 
devoted  to  QI  and  concluded  that  little  is  known, 
although  the  field  is  moving  rapidly.  Likewise  there 
is  not  much  information  about  the  cost  of  these 
activities;  data  are  lacking  on  fixed,  variable,  or 
opportunistic  costs,  as  well  as  on  the  savings  from 
QI  implementation. 

The  discussants  surmised  from  available  informa- 
tion that  training  of  individuals  in  QI  is  occurring  at 
multiple  levels  for  a  variety  of  people  in  health  care 
settings,  including  nurses,  graduate  students,  admin- 
istrators, physicians,  and  ancillary  health  personnel, 
and  in  many  different  educational  forms.  However, 
it  appears  there  are  no  organized  comparisons  of  the 
effectiveness  of  these  training  programs,  nor  any 


good  tools  to  measure  their  effectiveness.  A  number 
of  professional  organizations  are  involved  in  QI 
training  and  education,  including  the  Association 
of  University  Programs  in  Health  Administration, 
the  American  College  of  Physician  Executives,  the 
American  College  of  Medical  Quality,  the  National 
Association  of  Health  Care  Quality,  the  Institute  for 
Health  Care  Improvement,  and  many  others.  Again, 
however,  there  has  been  no  systematic  evaluation  of 
the  effectiveness  of  these  programs. 

Finally,  the  discussants  felt  that  the  Agency  for 
Health  Care  Policy  and  Research  (AHCPR), 
although  it  does  not  have  responsibility  for  profes- 
sional curriculum  reform,  could  influence  curricu- 
lums  through  such  mechanisms  as  conference  and 
dissertation  grants.  Four  research  questions  were 
identified: 

1.  Which  management  variables  involved  in  the 
organization  and  processes  of  care  have  an 
impact  on  care  outcomes,  and  how  are  they 
measured? 

2.  What  is  the  fund  of  knowledge  for  QI  and  for 
teaching  it? 

3.  AHCPR  may  be  able  to  sponsor  research  on 
how  to  improve  the  effectiveness  of  QI  educa- 
tion but  not  necessarily  provide  funds  for 
developing  a  curriculum  for  such  training. 

4.  What  models  or  delivery  systems  use  the  tenets 
of  QI  most  easily  and  why?  Are  their  outcomes 
better,  cheaper,  and  reproducible? 
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Workgroup  IV: 

Using  Quality  Improvement/Quality 
Assurance  Information 

Sheila  Leatherman,  Topic  Reporter 

Workshop:  Patients  as  Agents  for  QI/QA 

Facilitator:  Paul  Nutting,  M.D. 
Research  Discussant:  Lisa  Sullivan,  Ph.D. 

Workshop:  Purchasers  as  Agents  for  QI/QA 

Facilitator:  Jeanne  Keller 

Research  Discussant:  Bruce  N.  Davidson,  Ph.D. 

Workshop:  Influence  of  QI/QA  on 
Clinical  Decisionmaking 

Facilitator:  Doris  C.  Brooker,  M.D. 
Research  Discussant: 

Stephen  F.  Jencks,  M.D.,  M.P.H. 

Workshop:  Effects  of  Quality  Improvement/ 
Quality  Assurance  on  Health  Care  Systems 

Facilitator:  E.  David  Buchanan 
Research  Discussant:  Sheila  Leatherman 


The  workshop  on  patients  as  QI/QA  agents 
focused  on  ways  to  increase  the  patient's  participa- 
tion in  clinical  decisionmaking,  techniques  to 
enhance  patient-provider  communication,  and 
methods  to  learn  more  about  patients  and  their 
expectations  of  the  medical  encounter.  The  follow- 
ing topics  and  research  questions  were  identified: 

1.  Research  is  needed  on  restructuring  the  patient- 
provider  encounter  to  give  patients  a  more 
central  role  in  determining  the  course  of  their 
health  care.  Specific  research  questions  include: 
What  are  the  characteristics  of  patient-provider 
communication  that  lead  to  improved  outcomes? 
How  can  the  medical  encounter  be  structured  to 
enhance  effective  patient-provider  communica- 
tion? 

2.  We  need  to  identify  and  define  the  data  and 
information  currently  available  to  patients  and 
how  they  are  used.  Research  questions  in  this 
area  include:  How  do  patients  use  existing  data 
on  the  quality  and  effectiveness  of  health  care? 
How  do  they  sort  among  the  bewildering  array 
of  information  currently  available?  What  are  the 
effects  of  that  information  on  patients'  health- 


related  behaviors,  health  care-seeking  behavior, 
health  anxiety  levels,  and  satisfaction  with  care? 

3.  We  need  to  learn  more  about  patients'  values 
and  expectations  of  health  care  and  strategies 
that  might  provide  information  to  promote 
attainment  of  those  values  and  expectations.  The 
workgroup  identified  these  specific  research 
questions,  which  should  not  be  limited  to  single 
care  settings:  What  are  patients'  values  and 
expectations  in  health  care?  What  information 
do  they  need  to  make  informed  decisions?  What 
information  about  patients'  experiences  can  clin- 
icians and  provider  organizations  use  to  improve 
care?  Can  effective  strategies  be  developed  to 
provide  information  to  patients? 

Research  questions  developed  by  participants  in 
the  workshop  on  purchasers  as  agents  for  QI/QA 
reflected  their  awareness  that  the  effectiveness  of 
purchasers  depends  on  developing  a  partnership 
with  providers.  Since  QI  is  a  new  area  of  great  inter- 
est to  purchasers,  the  discussants  focused  on 
descriptive  research  to  provide  a  quick  initial  assess- 
ment of  the  state  of  the  art  and  develop  a  frame- 
work for  evaluating  and  communicating  results  that 
purchasers  have  achieved  in  QI/QA.  Three  research 
questions  were  identified: 

1.  Are  there  examples  of  existing  purchaser- 
provider  partnerships  for  achieving  QI/QA 
goals?  An  inventory  of  such  partnerships  and 
their  characteristics  is  desirable. 

2.  What  are  the  characteristics  of  these  purchaser- 
provider  partnerships?  Study  of  the  similarities 
and  differences  between  these  groups  could  pro- 
duce a  typology  of  such  partnerships. 

3.  What  is  the  relationship  between  particular  char- 
acteristics and  measures  of  success  in  achieving 
QI/QA  goals  in  such  partnerships?  For  example, 
is  success  in  achieving  the  goal  of  cost  savings  for 
the  purchaser  associated  with  providing  informa- 
tion about  quality  to  consumers? 

Participants  in  the  workshop  on  potential  uses  of 
QI/QA  information  to  enhance  clinical  decision- 
making stressed  the  importance  of  uncovering  the 
reasons  for  successes  and  failures  of  previous 
QI/QA  experiments.  They  also  discussed  the  need 
to  evaluate  and  compare  the  merits  of  various  sys- 
tems interventions.  The  following  research  needs 
were  identified: 
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1.  Retrospective  and  prospective  observational 
studies  of  natural  experiments  are  needed. 
Retrospective  studies  could  examine  areas  where 
there  have  been  successes — such  as  laparoscopic 
cholecystectomy  which  has  changed  practice 
patterns — as  well  as  failures,  as  in  screening 
practices  where  wide  variations  persist.  There  are 
opportunities  for  concurrent  prospective  studies 
in  many  natural  experiments  that  are  occurring 
around  the  dissemination,  proliferation,  and  use 
of  practice  guidelines.  These  guidelines  are  being 
used  by  commercial  insurers,  business  coalitions, 
Medicaid,  and  others. 

2.  Evaluations  are  needed  of  theories  underlying 
interventions  to  influence  clinical  decisionmak- 
ing at  the  systems  and  individual  practitioner 
levels.  Systems  interventions  that  should  be 
examined  and  compared  include  the  health 
education  model,  the  organizational  change 
model,  and  the  TQM/CQI  model. 

3.  Some  thought  should  be  given  to  convening  a 
conference  to  identify  which  organizational 
and  educational  factors  that  facilitate  QI/QA 
can  be  studied  and  what  methods  can  be  used 
to  study  them. 

The  workshop  on  the  effects  of  QI/QA  on 
health  care  systems  took  in  global  perspective  in 
developing  research  themes  by  viewing  QI/QA 
from  a  systems  viewpoint.  Interaction  among  all 
players  and  an  in-depth  look  at  previous  QI/QA 


efforts  were  stressed.  The  group  identified  the 
following  research  questions: 

1.  What  evidence  exists  on  the  effectiveness  of  QI 
components?  Descriptive  retrospective  research 
is  needed  to  examine  results  to  date,  using  both 
case  studies  and  anecdotal  information.  Evalua- 
tion research  is  also  needed,  with  emphasis  on 
prospective  studies,  on  whether  QI  and  QA 
produce  desired  results  and  what  the  unintended 
consequences  are.  These  studies  could  also  in- 
volve some  demonstrations. 

2.  What  are  the  essential  characteristics  of  the 
health  care  delivery  systems  that  have  imple- 
mented QI?  What  are  the  relative  contributions 
of  these  variables  to  the  successes  or  failures  of 
QI?  Examples  of  essential  characteristics  are 
management's  commitment  to  QI,  employee 
empowerment,  active  participation  of  physicians 
and  other  health  care  providers,  types  of  delivery 
systems  (hospitals,  managed  care,  long-term 
care),  and  the  populations  served. 

3.  Can  QI  contribute  to  fostering  integrated  mod- 
els of  community  health  service  delivery?  Does 
QI  provide  common  measures  and  tools  for 
linking  all  "players"  (providers,  purchasers, 
policymakers,  and  so  forth)?  Can  QI  make  a 
contribution  to  the  community  by  improving 
the  health  status  of  individuals  or  populations, 
lowering  the  cost  of  medical  care,  and  providing 
decisionmaking  support  to  the  community? 
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Appendix  A 


Users  of  Quality  Improvement/Quality  Assurance  Information: 
Panel  and  Floor  Discussion 


The  following  panel  discussion  responded  to 
opening  addresses  at  the  conference  and  was 
intended  to  set  the  stage  for  workshop  deliberations 
and  presentations  during  the  conference.  Panelists 
focused  on  the  users  of  quality  improvement/quality 
assurance  (QI/QA)  information  and  discussed  the 
components  of  a  research  agenda  that  would  have 
practice  relevance,  based  on  their  views  of  the 
QI/QA  needs  of  their  constituent  groups. 

Rodney  Dueck,  M.D.  The  following  are  research- 
able  QI/QA  questions:  What  is  the  technical  quality 
of  care  that  is  being  provided?  How  does  that  qual- 
ity vary?  How  do  diagnosis,  evaluation,  and  treat- 
ment processes  vary?  How  do  outcomes  vary?  Do 
the  leaders  of  health  care  institutions  understand 
that  quality  means  more  than  just  technical  quality, 
and  that  quality  of  service  is  also  important?  How 
large  is  the  role  of  fear  in  unwillingness  to  report 
abnormalities  or  in  ordering  unnecessary  tests? 
How  good  is  access  to  medical  care?  Are  medical 
care  workers  attuned  to  the  needs  of  patients  as  cus- 
tomers? Do  they  understand  their  customers?  Are 
patients  treated  with  respect,  and  are  their  ques- 
tions answered?  Do  patients  participate  in  decisions 
affecting  their  care?  How  can  we  respond  to  such 
challenges  while  reducing  costs  and  maintaining 
quality?  Do  the  leaders  in  health  care  institutions 
understand  that  their  employees  need  to  feel  appre- 
ciated, to  have  a  say  in  their  work,  and  to  experi- 
ence joy  in  it?  Does  management  rely  on  finger- 
pointing  for  quality  improvement,  or  is  there  a 
process  of  collegial  trust  and  sharing? 

Dr.  Dueck  is  the  medical  director  of  the  Park  Nicollet  Medical 
Center. 

Mary  Jo  Gibson,  M.A.  A  key  issue  in  setting  a 
research  agenda  is  the  lack  of  controlled  studies  to 
determine  whether  continuous  quality  improvement 
(CQI)  really  works.  Another  research  priority 
should  be  the  use  of  patients'  perspectives  as  out- 
come measures — for  example,  quality  of  life  and 


perception  of  pain.  In  treatment,  the  preferences  of 
patients  should  have  priority,  and  the  satisfaction  of 
patients  requires  moving  beyond  a  focus  on  the 
"hotel"  aspects  of  the  hospital  setting.  There  is  a 
need  to  examine  communication  processes  between 
providers  and  patients  and  to  assess  the  specific 
needs  and  expectations  of  patients.  There  is  also  a 
need  to  examine  the  quality  of  care  in  home  and 
community  settings  as  determinants  of  outcomes  for 
outpatients  and  patients  with  chronic  illness.  How  do 
these  different  care  settings  influence  both  QA  and 
QI?  Another  critical  issue  is  protection  of  patients' 
privacy,  since  linkage  of  health  data  bases  will  be 
necessary  for  cost  analysis  and  other  health  services 
research.  Research  is  needed  to  develop  safeguards 
against  improper  access  to  this  information. 

Ms.  Gibson  is  a  senior  analyst  at  the  Public  Policy 
Institute  of  the  American  Association  of  Retired  Persons. 

Robert  Hungate,  M.B.A.  In  the  future  the  patient 
will  be  the  one  who  drives  the  system;  the  profes- 
sional will  no  longer  be  the  controller.  There  is  a 
striking  contrast  between  what  is  reported  in  out- 
comes research  studies  and  public  perceptions 
about  outcomes.  QI  is  not  a  new  idea;  the  system 
has  been  trying  to  improve  quality  of  care  for  a  long 
time.  What  is  changing  is  the  degree  of  accountabil- 
ity for  results,  and  the  change  is  dramatic.  The 
change  is  being  driven  partly  by  the  news  media,  but 
essentially  it  is  based  on  the  need  to  provide  inter- 
nal control  mechanisms.  What  is  evolving  is  an 
enhancement  of  the  professional  model  where  the 
core  strength  remains  the  professional-patient  rela- 
tionship, but  this  is  further  strengthened  by  better 
information  technology  support.  The  perceptions  of 
patients  are  a  crucial  element  of  quality  and  must  be 
built  into  the  system.  Communication  between 
providers  and  patients  is  critical  and  must  be 
focused  on  what  the  patient  needs  and  expects. 


Mr.  Hungate  is  the  president  of  the  Physician/ Patient  Partner- 
ships for  Health. 
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Pamela  Waggoner,  R.N.,  M.S.N.  As  manager  of 
an  acute  care  facility  and  a  lecturer  on  CQI,  I  am 
frequently  told  by  members  of  my  audience  that  the 
concept  is  interesting,  but  they  have  difficulty  seeing 
its  relevance  to  their  work.  There  are  frequent 
questions  along  the  lines  of,  "How  will  CQI  help  me 
in  the  intensive  care  unit  at  2:00  a.m.  when  a  patient 
is  crashing?"  "How  does  CQI  help  me  do  a  better 
job  in  taking  care  of  my  patients?"  The  basic  ques- 
tion is,  what  are  the  pragmatic  tools  of  CQI  and  do 
they  exist  globally  or  must  they  be  derived  locally? 
Answering  this  question  requires  formulating 
research  questions  that  focus  on  the  partnership 
between  caregivers  and  patients.  When  hospitals 
begin  to  look  at  that  relationship,  we  will  begin  to 
see  the  cost-effective,  quality  care  patients  have 
been  expecting.  Another  research  question  to  be 
addressed  is  how  CQI  can  improve  planning  for 
staffing  needs,  and  how  the  cost  and  resources 
required  for  a  certain  level  or  type  of  care  can  be 
assessed  effectively? 

Ms.  Waggoner  is  the  director  of  Quality  Management, 
DePaul  Health  Center. 

Kathleen  N.  Lohr,  Ph.D.  Questions  about  informa- 
tion technology,  the  use  of  large  aggregate  data 
sets,  and  concern  about  confidentiality  are  about 
to  become  major  topics  of  public  debate.  The 
Institute  of  Medicine  is  undertaking  a  study  on  these 
issues,  and  it  is  already  evident  that  the  key  issues 
are  related  to  confidentiality.  Reliability  and  validity 
are  secondary  concerns.  It  appears  that  protection 
for  data  banks  in  such  matters  as  human  immuno- 
deficiency virus  infection  and  mental  health  is  not 
as  good  as  we  might  think. 


Ms.  Lohr  is  the  deputy  director  of  the  Division  of  Health  Care 
Services  at  the  Institute  of  Medicine. 

Floor  Discussion 

Q:  Are  there  are  any  situations  where  studies  of 
outcome  data  have  actually  been  used  to 
improve  the  quality  of  care?  Have  studies  shown 
an  effect? 

A:  [Dr.  Greenfield  J  I  am  not  aware  that  it  has  been 
done  yet,  although  a  number  of  institutions  are 
attempting  to  incorporate  information  from 
outcome  studies  to  improve  quality  of  care. 
Publication  of  findings  from  some  of  these 
efforts  is  imminent. 


Q:  Does  the  need  to  satisfy  both  internal  and 
external  customers  have  any  connection  to  the 
"outsider"  status  of  medical  staff  in  hospitals? 
Should  health  care  professionals  also  be  treated 
as  customers  so  conditions  can  be  changed  to 
make  them  better  able  to  take  care  of  patients? 

A:  [Dr.  Smits]  The  problem  of  treating  physicians 
as  customers  is  that  they  are  intimately  involved 
in  the  processes,  and  there  is  a  possibility  of  con- 
flicts arising  between  what  works  well  for  them 
and  what  works  well  for  the  patients.  As  partici- 
pants in  the  care  process,  physicians  are  a  neces- 
sary part  of  improving  it.  However,  there  are  sit- 
uations where  physicians  are  the  customers  and 
need  to  be  dealt  with  as  such.  A  good  example 
is  surgeons  who  find  institutional  delays  an 
impediment  to  proper  care.  A  virtue  of  the  CQI 
concept  is  that  it  disallows  blaming  and  instead 
promotes  working  with  others  to  improve 
situations.  In  many  cases,  when  a  situation  is 
analyzed,  it  is  clear  that  no  individual  is  at  fault; 
the  problem  lies  in  the  processes.  Like  patients, 
those  in  hospitals  who  are  involved  in  patient 
care  are  not  always  treated  well  as  customers 
and  are  beginning  to  see  the  situation  in  those 
terms.  Changing  the  situation  will  require  time. 

Q:  Does  Dr.  Smits  suggest  that  researchers  are 
disenchanted  with  the  Uniform  Clinical  Data 
Set  (UCDS)  that  is  used  by  peer  review 
organizations? 

A:  [Dr.  Smits]  There  are  concerns  about  the  sheer 
size  of  the  UCDS,  and  it  is  doubtful  that  a  data 
set  that  large  is  accurate.  A  more  parsimonious 
data  set  containing  the  right  data  elements  is 
more  valuable  and  would  put  fewer  demands  on 
hospitals.  If  too  much  information  is  collected  it 
is  not  likely  to  be  collected  well. 

Q:  Has  there  been  any  improvement  in  technical 
care  and  interpersonal  care? 

A:  [Dr.  Greenfield]  A  number  of  people  are  work- 
ing to  make  interpersonal  care  as  systematic  as 
technical  care.  In  many  respects  the  measure- 
ment of  interpersonal  care  is  more  valid  and  bet- 
ter refined  than  many  of  our  medical  measure- 
ments. Interest  in  interpersonal  care  issues  is 
considerable.  Several  studies  are  giving  equal 
weight  to  technical  and  interpersonal  care  in 
their  impact  on  outcomes.  In  the  case  of  chronic 
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illnesses,  interpersonal  care  may  have  a  greater 
effect  on  outcomes  than  technical  care. 

Q:  Patients  entering  health  care  typically  are 
assessed  for  such  things  as  pulse,  respiration, 
weight,  and  blood  metabolites,  but  they  are  sel- 
dom examined  for  conditions  such  as  depression, 
intoxication,  or  availability  of  transportation, 
which  are  being  seen  as  increasingly  relevant. 
Will  consideration  and  measurement  of  the 
latter  factors  ever  become  a  standard  part  of 
medical  practice  as  valid  clinical  data  that  should 
be  part  of  the  clinical  record?  If  so,  what  will  it 
take  to  bring  that  about? 

A:  [Dr.  Greenfield]  One  way  to  bring  it  about  is  to 
publish  data  on  these  factors  in  medical  journals 
to  bring  them  to  the  attention  of  physicians  and 
ultimately  to  administrators.  Studies  in  these 
areas  are  a  potential  contribution  to  medicine, 
but  the  quality  of  measurements  for  some  of 
these  factors  need  improvement. 

[Dr.  Smits]  It  is  surprising  that  so  many  reports 
on  cataract  surgery  have  dealt  with  matters  such 
as  infection  rates,  but  not  on  the  question  of  how 
well  patients  could  see  and  the  quality  of  their 
lives.  It  is  important  to  develop  easy  ways  to 
measure  such  factors.  There  are  good  measure- 
ments for  depression  but  not  for  such  things  as 
whether  home  and  lifestyle  factors  are  helping 
or  hindering  treatment  success.  Elaborate  check- 
sheets  to  address  these  kinds  of  issues  are  not 
very  practical.  Simpler  screening  tools  are 
needed  for  routine  use,  with  the  more  elaborate 
tools  used  only  when  they  are  relevant. 

Q:  In  organizational  theory,  a  distinction  between 
administrative  and  clinical  decisionmaking  is 
relevant  to  points  made  by  Dr.  Smits  about 
problems  arising  from  rigidity  in  professional 
roles.  Can  a  focus  on  quality  help  us  overcome 
the  damaging  effects  of  having  two  decision- 
making processes  in  our  health  care  organiza- 
tions, one  administrative  and  the  other  clinical, 
and  with  neither  side  understanding  the  needs 
of  the  other? 

A:  [Dr.  Smits]  CQI  offers  systematic  opportunities 
to  deal  with  that  issue,  since  it  involves  physi- 
cians and  administrators  working  together  to 
meet  the  health  needs  of  patients.  It  is  doubtful 
that  administrators  in  these  times  are  not  sensi- 
tive to  the  requirements  of  the  care  process,  and 


many  physicians  have  expressed  satisfaction 
about  the  results  obtained  from  CQI. 

Q:  Dr.  Greenfield  suggested  that  making  a  differ- 
ence requires  one  clinician  to  challenge  the  way 
another  clinician  practices.  How  can  such 
barriers  to  changing  physicians'  behavior  be 
overcome? 

A:  [Dr.  Greenfield]  One  possible  way  of  dealing 
with  such  issues  is  with  managed  care,  which 
puts  pressure  on  people  to  debate.  Physicians 
have  to  talk  to  and  challenge  each  other.  If  they 
do  not  do  this  on  their  own,  the  alternative  is 
likely  to  be  administrative  regulatory  procedures 
that  physicians  find  onerous.  It  might  be  neces- 
sary to  excuse  older  physicians  from  participat- 
ing in  such  encounters  and  concentrate  on  the 
newer  physicians. 

[Dr.  Smits]  My  experience  is  not  consistent  with 
the  idea  that  people  cannot  change.  I  know  a 
surgeon  who  invited  his  colleagues  to  study  his 
cases  after  he  observed  an  increase  in  complica- 
tion rates.  The  colleagues  did  so  and  found  the 
cause  of  the  problem.  Objective  data  and  a 
focus  on  system  problems  are  much  better  than 
a  focus  on  individual  physician  problems.  There 
are  a  lot  of  system  problems  in  hospitals  that 
can  be  fixed  without  having  to  blame  an  individ- 
ual, although  it  is  necessary  to  be  tough  when 
an  individual  is  at  fault. 

Q:  Is  there  a  role  for  qualitative  research  in  such 
quality  of  care  issues  as  defining  roles  and 
behavior  and  changing  attitudes? 

A:  [Dr.  Greenfield]  There  is  indeed  a  role  for  quali- 
tative studies.  We  are  in  a  period  of  quantitative, 
reductionist,  and  overly  technical  analysis,  and 
more  balance  is  needed.  We  need  to  keep  in 
mind  that  much  of  the  quantitative  work  being 
done  owes  its  beginning  to  qualitative  and 
observational  work  that  was  not  even  published. 

Q:  What  are  the  greatest  concerns  for  quality  of 
health  care  and  what  are  the  prospects  for  the 
rest  of  the  century? 

A:  [Dr.  Dueck]  Health  care  will  comprise  20  percent 
of  the  gross  national  product  by  the  year  2000, 
and  this  is  a  trend  that  is  unsustainable.  Deci- 
sionmaking is  in  a  paralytic  state.  I  am  less  con- 
cerned about  whether  physicians  will  or  will  not 
change  because  I  have  seen  many  of  them 
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change  and  do  it  gladly.  My  concern  is  with  the 
capacity  to  make  the  decision  for  CQI.  Medical 
residents  I  have  lectured  about  CQI  show  very 
little  interest  or  awareness  of  the  subject.  Prac- 
ticing physicians  are  much  the  same.  The  lack  of 
knowledge  about  management,  CQI  theory,  and 
its  use  in  health  care  corporations  is  disturbing. 
CQI  requires  sustained  enthusiasm  for  5  to  10 
years  to  produce  change.  People  who  lack  knowl- 
edge reject  ideas  and  resist  change.  Dr.  Green- 
field described  several  studies  that  show  the 
merit  of  CQI,  yet  nothing  changes;  we  need  to 
ask  why.  This  leads  to  further  research  questions: 
What  are  leaders  doing  to  educate  themselves 
about  CQI?  To  what  degree  do  medical  schools 
incorporate  it  in  their  training?  Do  management 
leaders  understand  it  and  their  role  in  CQI? 

Q:  Given  the  limited  resources  at  Agency  forHealth 
Care  Policy  and  Research  and  the  research 
community  in  general,  what  are  the  high-priority 
research  topics  for  CQI? 

A:  [Ms.  Waggoner]  About  50  percent  of  health  care 
institutions  will  embark  on  CQI  initiatives  over 
the  next  3  to  5  years.  The  alarming  fact  is  that 
about  90  percent  of  those  initiatives  will  fail.  In 
my  institution,  when  we  look  at  an  investment 
of  hundreds  of  thousands  of  dollars  in  CQI  and 
know  it  could  fail,  we  want  information  on  suc- 
cessful methodologies.  So  I  think  there  is  a  need 
for  research  on  hospital  cultures.  The  process 
changes  in  CQI  are  not  individualistic,  yet  we 
have  independent  physicians  in  the  institution. 
When  we  looked  at  our  proposal  options,  we 
found  enthusiasm  but  no  data  on  successes  to 
provide  guidance.  Studies  of  CQI  successes  and 
failures  are  needed  so  people  can  learn  from  them. 

Q:  How  might  we  think  about  consortia  and  support 
of  research  on  quality  among  sectors?  How 
might  they  work  and  communicate?  Could 
corporations  help  fund  this? 

A:  [Mr.  Hungate]  Benefit  managers  do  try  to 
provide  good  health  benefits  for  employees  but 


are  limited  primarily  to  plan  design  in  their 
ability  to  do  so.  Employees  cannot  make  valid 
decisions  about  health  care  needs  during  open 
enrollment  unless  they  have  a  preexisting 
condition  and  knowledge  of  how  plans  manage 
that  condition.  Open  enrollment  does  let  people 
move  from  one  plan  to  another,  but  they  usually 
do  not  study  these  changes.  This  could  be 
researched  to  help  design  better  benefit  plans 
that  would  stimulate  QI. 

Q:  What  is  the  difference  between  CQI,  total 
quality  management  (TQM),  and  total  quality 
improvement?  Are  they  the  same  thing,  and 
does  it  matter? 

A:  [Dr.  Dueck]  These  acronyms  denote  various 
corporate  responses  to  quality  issues.  What  they 
all  have  in  common  is  an  orientation  to  the  cus- 
tomer, taking  time  to  understand  the  customer, 
and  working  continuously  to  align  processes  to 
the  customer's  needs  and  expectations.  Another 
thing  they  have  in  common  is  actions  to  bring 
about  cooperation,  collaboration,  absence  of 
fear,  and  a  sense  of  teamwork  and  trust  among 
caregivers.  They  also  share  statistical  thinking, 
process-oriented  analytical  problem  solving, 
and  application  of  the  scientific  method  to 
decisionmaking. 

[Dr.  Palmer]  The  concepts  overlap.  TQM  is 
more  a  strategy  for  quality  and  implies  cultural 
and  organizational  change.  CQI  deals  more 
with  the  actual  business  of  improvement. 

Q:  In  looking  at  quality,  client  satisfaction  is  a  main 
focus.  My  question  is  about  methodology.  Are 
different  methodologies  needed  for  surveying 
clients  for  level  of  satisfaction  when  the  care  is 
in  the  home? 

A:  [Dr.  Lohr]  The  Institute  of  Medicine's  study 
of  quality  assurance  in  Medicare  considered 
home  health  care  a  high  priority  area.  The 
methods  and  the  questions  to  be  asked  in  home 
health  care  are  different,  because  it  is  a  distinct 
population  with  a  distinct  set  of  problems. 
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Appendix  B 


Reaction  to  Workshop  Recommendations  for  Research  Priorities: 
Panel  and  Floor  Discussion 


The  members  of  this  panel  reacted  to  the  pre- 
sentations and  research  priorities  identified  in  the 
workshop  reports. 

Pamela  Waggoner,  R.N.,  M.S.N.,  C.C.R.N. 

A  healthy  skepticism  was  evident  in  the  conference 
discussions  about  continuous  quality  improvement 
(CQI).  The  health  care  field  has  embraced  the  con- 
cept of  continuous  improvement  in  the  past  year 
and  a  half  and  appears  willing  to  invest  a  lot  of 
money  in  a  process  that  so  far  has  not  been  vali- 
dated by  rigorous  scientific  inquiry.  On  a  more 
promising  note,  representatives  of  various  health 
care  interest  groups  demonstrated  a  collaborative 
spirit  and  interest  in  working  toward  continuous 
improvement  in  health  care. 

Although  the  conference  participants  did  an 
impressive  job,  with  much  discussion  about 
processes  of  health  care  delivery,  a  number  of 
important  questions  and  issues  merit  further  investi- 
gation. For  example,  we  should  consider  the  process 
of  identifying  specific  elements  of  care  used  during 
an  episode  of  illness.  We  need  to  ask,  "How  does 
the  sequence  of  specific  care  activities  affect  out- 
come? Can  continuous  improvement  affect  episodes 
of  care  to  the  same  extent  that  it  can  affect  episodes 
of  illness?"  The  outcomes  are  not  the  same  for  an 
episode  of  care  and  an  episode  of  illness,  and  it  is 
possible  that  continuous  improvement  could  affect 
one  more  than  the  other. 

Another  important  question  is  which  methods 
can  accelerate  the  processes  of  CQI,  a  transforma- 
tion that  now  can  take  5  to  10  years.  It  is  not  clear 
that  the  health  care  industry  has  5  to  10  years  to 
achieve  this  transformation.  Are  there  ways  to 
accelerate  the  process? 

Many  efforts  to  implement  CQI  in  components 
of  the  health  care  industry  seem  to  fail  after  about 
18  months  to  3  years.  Can  these  failures  be  pre- 
dicted, and  can  steps  be  taken  to  prevent  failure? 
Can  the  health  care  industry  learn  anything  from 
similar  failures  that  have  occurred  in  other  kinds  of 
industries? 


Several  of  the  workshops  gave  attention  to  the 
most  effective  ways  to  use  continuous  improvement 
data,  not  only  to  change  care  practices  but  also  to 
change  expectations  of  consumers  and  payers.  A 
question  needing  further  investigation  is  whether 
outcomes  data  should  be  used  as  tools  for  negotiat- 
ing preferred  provider  organization  and  health 
maintenance  organization  contracts  in  acute-care 
and  long-term  care  settings.  An  important  topic  that 
was  not  addressed  by  any  of  the  work  groups  con- 
cerns the  impact  of  CQI  on  the  industries  that 
design  the  technologies  of  medical  care. 

Specific  patient  populations  were  considered  by 
discussants  in  the  context  of  CQI,  but  it  would  have 
been  helpful  if  the  role  of  hospitals  also  had  been 
considered.  Hospitals  need  to  take  specific  patient 
populations  into  account  in  implementing  CQI  and 
in  designing  outcomes  measures,  practice  guide- 
lines, and  evaluation  criteria.  They  also  need  to 
examine  delivery  processes,  review  criteria,  and  out- 
come criteria  for  high  volume,  high-risk  activities 
such  as  coronary  bypass  surgery  and  total  hip 
replacement. 

Another  area  that  merits  further  attention  is  the 
impact  of  comorbidity  on  outcomes.  How  can 
guidelines  account  for  the  impact  of  comorbidity  on 
outcomes? 


Ms.  Waggoner  is  Director,  Quality  Management,  DePaul  Health 
Center,  Daughters  of  Charity  National  Health  System 

Robert  Hungate,  M.B.A.  From  the  perspective  of 
health  care  purchasers,  it  is  clear  that  health  care  is 
on  the  threshold  of  profound  changes.  There  is  pal- 
pable tension  in  the  purchaser  community  about  the 
state  of  the  health  care  industry.  Frustrations  are  in- 
tense and  are  growing.  The  need  for  improvement  is 
unquestionable  when,  for  example,  the  rate  of  cesar- 
ean sections  is  so  variable  geographically  and  gener- 
ally is  judged  to  be  excessive  in  most  places.  Such 
shortcomings  have  been  apparent  for  a  long  time. 
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From  the  perspective  of  patients,  a  number  of 
recommendations  from  the  workshops  merit  strong 
endorsement  and  support.  The  recommendations 
on  use  of  information  technology  for  quality  assur- 
ance and  quality  improvement  (QI)  merit  support. 
We  do  not  always  need  new  information;  rather  we 
must  learn  how  to  use  the  information  we  already 
have  more  effectively. 

It  is  always  surprising  to  hear  recommendations 
for  research  on  why  people  do  things.  The  answer 
seems  obvious — people  change  for  their  self  interest, 
and  they  adopt  practices  that  they  think  will  make 
their  lives  better.  I  come  from  an  industry  that  has 
marketed  information  systems,  and  it  is  clear  that 
people  do  not  buy  something  unless  they  believe  it  is 
better  than  what  they  have.  Once  people  come  to 
realize  that  something  can  make  their  lives  better,  no 
research  is  needed  on  how  to  disseminate  informa- 
tion about  it.  It  simply  happens.  Laparoscopic 
cholecystectomy  is  a  good  example.  The  way  to 
get  something  disseminated  is  to  make  it  good. 
Tinkering  at  the  margins  is  not  what  is  needed.  What 
is  needed  is  real  action  on  things  that  will  make  a 
difference,  such  as  improving  information  systems. 

With  regard  to  the  recommendations  for  educa- 
tion, the  new  generation  that  will  take  over  the 
health  care  industry  could  be  called  "the  Nintendo 
generation,"  and  it  has  a  very  different  set  of  skills 
and  behaviors  than  those  who  run  it  now.  In  15 
years,  they  will  be  using  the  tools  that  are  evolving 
now,  and  our  task  is  to  facilitate  that  transition. 
Thus,  education  for  them  is  of  major  importance. 

The  recommendations  concerning  patients  as 
agents  for  QI  also  are  of  major  importance,  for  it  is 
in  the  area  of  the  doctor-patient  relationship  where 
major  improvement  in  quality  of  care  will  take  place. 

There  is  a  need  to  shift  our  focus  from  health 
care  to  health  itself.  Health  can  be  measured  in 
populations;  this  is  the  standard  public  health 
approach.  The  purchaser  community  continues  to 
concentrate  on  health  care  costs,  however,  not  on 
the  health  of  the  work  force.  The  health  of  the 
work  force  is  actually  more  critical  to  an  employer 
than  health  care  costs.  There  is  much  more  to  be 
lost  from  lack  of  motivation  in  unhealthy  workers 
than  from  the  cost  of  health  care.  An  increased 
focus  on  community  health  can  help  shift  the  focus 
of  health  care  purchasers. 

Mr.  Hungate  is  President,  Physician/Patient  Partnerships  for 
Health. 


Mary  Jo  Gibson,  M.A.  Quality  of  care  is  an 
important  component  of  a  health  care  reform  pro- 
posal being  prepared  by  the  American  Association 
of  Retired  Persons  (AARP).  AARP  is  interested  in 
health  care  reform  for  the  benefit  of  all  Americans, 
not  just  its  own  members.  Its  proposal  includes 
universal  coverage,  cost  containment,  quality 
assurance,  and  long-term  care. 

Dr.  Leatherman's  concern  about  people  outside 
the  health  care  system  is  well  founded.  Patients  and 
purchasers  were  considered  as  agents  of  change  in 
workshop  discussions,  but  there  are  36  million 
Americans  without  health  insurance,  and  issues  of 
quality  of  care  alone  are  somewhat  irrelevant  to 
them.  There  are  also  at  least  20  million  more  Amer- 
icans who  are  underinsured.  The  needs  of  this  large 
group  should  be  studied  as  a  quality  of  care  issue. 

In  the  area  of  information  technology,  workshop 
emphasis  on  the  roles  of  technology,  policy,  legisla- 
tion, and  public  opinion  in  protecting  privacy  and 
confidentiality  is  consonant  with  the  AARP's 
concerns.  Protecting  the  privacy  of  data  base  infor- 
mation is  an  important  area  for  research.  Canada 
and  other  countries  have  used  unique  patient  identi- 
fiers in  data  bases  for  some  time,  and  it  may  be 
worthwhile  to  study  their  systems  to  determine  how 
well  they  protect  privacy.  Another  highly  salient 
question  emerging  from  the  information  technology 
sessions  was  how  to  incorporate  patient-specific  val- 
ues and  preferences  into  decision  support  systems. 

The  discussion  sessions  on  methods  and  meas- 
ures examined  the  question  of  longitudinal  measures 
needed  to  follow  patients  across  care  settings. 
Research  on  tracking  patients  through  the  whole 
system — including  post-acute  and  long-term  care  as 
well  as  acute  care — should  be  given  high  priority. 

The  session  on  control  and  responsibility  for 
quality  review  raised  questions  with  many  policy 
implications.  Of  particular  interest  is  the  relation- 
ship of  State  and  Federal  regulatory  activities  to  QI. 
A  number  of  important  questions  were  raised  in  the 
session.  What  are  the  effects  of  oversight?  In  what 
settings  or  circumstances  should  existing  inspection 
methods  be  retained?  How  does  external  review 
stimulate,  support,  or  subvert  the  QI  process? 

It  was  also  pleasing  to  see  the  workshop  focus 
on  whether  the  CQI  process  works  and  whether  it 
produces  unintended  adverse  consequences.  The 
recommendation  to  study  restructuring  of  the 
patient-provider  encounter  to  improve  communica- 
tion was  appropriate.  However,  it  is  also  important 
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to  examine  the  effects  of  differing  payment  and 
delivery  systems  (such  as  various  forms  of  managed 
care)  on  this  encounter,  as  well  as  how  the  charac- 
teristics of  physicians  and  patients  influence  the 
encounter.  Finally,  drawing  upon  research  in  fields 
such  as  education  and  psychology  might  help  us 
better  understand  and  influence  both  physician  and 
patient  behaviors. 


Ms.  Gibson  is  Senior  Analyst,  Public  Policy  Institute,  American 
Association  of  Retired  Persons. 


Rodney  R.  Dueck,  M.D.  Two  important  themes 
emerged  during  the  conference.  First  is  the  impor- 
tance of  continued  learning.  Much  of  what  we  need 
to  learn  about  QI  in  health  care  exists  in  literature 
other  than  that  usually  studied  by  health  care 
professionals.  Familiarity  with  that  literature  may 
provide  shortcuts  to  useful  knowledge.  Second  is 
the  concept  of  systems  thinking.  Whether  we  use  a 
public  health  perspective  or  focus  on  the  commu- 
nity, we  are  thinking  in  terms  of  the  whole.  In 
contrast,  optimizing  the  parts  at  the  expense  of  the 
system  as  a  whole  could  be  very  costly.  Our  para- 
digms of  individuality  and  entrepreneurship  are 
valuable  and  powerful,  but  they  also  can  blind  us  to 
the  value  and  power  of  cooperation  and  systems 
thinking.  According  to  Deming's  theory  of  manage- 
ment, optimization  of  the  parts  at  the  expense  of 
the  whole  leads  to  decline — optimization  of  the 
whole  benefits  all,  over  the  long  term. 

Several  of  the  workshop  recommendations  are 
of  direct  interest  to  physicians.  In  particular, 
physicians  are  likely  to  be  very  receptive  to  the 
idea  of  a  decision  support  system.  Although  it  is 
important  not  to  diminish  professional  autonomy  in 
caring  for  a  patient,  it  also  is  important  to  reduce 
the  incidence  of  unintended  variation  in  providing 
care.  Physicians  want  to  do  a  good  job  and  are  not 
averse  to  being  helped  by  a  support  system. 

The  recommendation  to  examine  QI  successes 
and  failures  in  the  real  world,  and  to  learn  from 
them,  is  especially  valuable.  Understanding  the 
causes  of  success  and  failure  can  accelerate  the 
process  of  instituting  CQI. 

In  the  area  of  quality  review  criteria,  the  most 
important  element  is  the  nature  of  the  relationship 


between  reviewers  and  those  being  reviewed.  The 
process  should  be  nonaccusatory  and  educational, 
and  it  should  focus  on  improving  the  system  as  a 
whole  and  not  on  finding  individuals  to  blame. 
Accountability  is  important,  but  it  is  most  powerful 
when  it  is  voluntary,  and  a  teaching  and  support 
mode  will  lead  to  more  receptive  providers. 

In  developing  standards,  it  is  important  that  the 
standards  be  credible  and  respectable.  Pulse  oxime- 
try in  general  anesthesia  is  an  excellent  example  of 
a  standard.  It  has  a  powerful  impact  on  the  lives  of 
patients  by  reducing  complication  rates  and  thus 
gives  credibility  to  the  agency  or  institution  that 
enforces  it.  Vague  standards,  which  are  perceived  as 
arbitrary  and  capricious,  create  confusion,  resent- 
ment, and  loss  of  movement. 

Another  research  question  developed  during 
the  conference  concerns  the  effects  of  regulatory 
oversight  on  QI.  Oversight  and  QI  are  not  neces- 
sarily incompatible  if  there  is  trust  between 
providers  and  regulators,  but  very  little,  if  any,  trust 
exists  between  them  at  this  time.  The  best  approach 
to  overcoming  this  problem  is  teaching,  improving 
systems,  and  asking  for  voluntary  cooperation. 
Measurement  and  feedback  without  threat  of 
reprisal  can  be  very  helpful. 

Regarding  the  recommendation  to  study  the 
experiences  of  other  industries  in  QI,  an  enormous 
literature  base  exists  on  this  that  could  be  adapted 
to  the  health  care  industry  and  medical  thinking. 

On  patients  as  agents  for  change,  it  is  critical 
that  physicians  learn  to  understand  the  views  of 
patients.  Anything  that  can  be  done  to  facilitate 
that  understanding  is  a  crucial  first  step.  The 
second  step  is  for  physicians  to  examine  how  their 
processes  are  discordant  with  the  thinking  and 
expectations  of  patients. 

Key  issues  in  the  area  of  purchasers  as  agents 
for  change  are  the  concepts  of  partnership  and 
optimizing  the  system  as  a  whole.  Other  industries 
are  ahead  of  the  health  care  industry  in  implement- 
ing these  concepts. 

As  others  have  noted,  the  problem  of  people 
without  health  insurance  is  major  and  must  be 
solved.  Its  solution  may  come  through  development 
of  a  systems  approach  and  a  cooperative  spirit  in 
the  health  care  industry. 

Dr.  Dueck  is  Medical  Director,  Park  Nicollet  Medical  Center. 
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Floor  Discussion 

Q:  A  recurring  theme  in  this  conference  is  the  need 
for  access  to  data  on  health  care  quality.  Peer 
review  organizations  (PROs)  in  every  State 
collect  and  analyze  large  amounts  of  information 
on  health  care  quality,  although  they  tend  to  use 
it  to  identify  bad  performers.  Could  better  use 
be  made  of  PRO  information  resources  for  CQI, 
and  could  the  PROs  themselves  evolve  into 
instruments  for  CQI? 

Discussion  among  the  panelists  and  members  of 
the  audience  brought  out  the  following  points: 

•    Many  PROs  have  been  moving  from  a  focus  on 
punitive  action  toward  use  of  data  for  improving 
care.  As  the  health  care  industry  in  general 
moves  toward  QI,  it  will  not  be  necessary  to 
"twist  the  arms"  of  the  PROs  to  shift  their  focus. 


•    In  addition,  there  has  been  a  trend  in  recent 
years  for  PROs  to  get  involved  in  research  on 
technology  assessment  and  quality  of  care, 
suggesting  that  they  could  become  an  important 
resource  for  CQI.  The  Agency  for  Health  Care 
Policy  and  Research  is  promoting  this  transition 
from  affixing  blame  to  surveillance  and  educa- 
tion. A  key  question,  however,  is  whether  it  is 
possible  for  an  external  organization  such  as  a 
PRO  to  relate  to  providers  in  an  educational 
manner,  and,  if  so,  what  would  be  necessary  to 
accomplish  this.  The  existence  of  PROs  in  every 
State,  each  doing  things  in  different  ways,  does 
create  opportunities  for  "natural  experiments" 
which  could  help  to  answer  that  question. 
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